00000100:00080000:3.0F:1769075020.772640:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075020.772655:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1769075020.772659:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075020.772663:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1769075020.772666:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075020.772670:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1769075020.772674:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075020.772678:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1769075020.772681:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075020.772685:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1769075020.836686:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075005/real 1769075005] req@ffff969e0a7fca80 x1855007923559680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075020.836719:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923559680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075020 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075020.836737:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:0.0F:1769075020.836779:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075020.836782:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075020.836786:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923559680:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075020.836964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560320:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075021.860702:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075021.860714:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1769075021.860718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075021.860722:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1769075021.860725:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075021.860729:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1769075021.860732:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075021.860749:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1769075021.860751:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075021.860753:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0F:1769075021.860761:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075022.884829:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075022.949344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075020/real 1769075022] req@ffff969e10b61f80 x1855007923560320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075022.949379:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923560320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075022.949397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075022.949404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075022.949410:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560320:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769075023.908769:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075024.932804:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075025.892975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560448:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769075025.956825:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769075026.021066:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.204.143:988 00000400:00000100:0.0:1769075026.021135:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769075026.021162:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075026.021374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075025/real 1769075026] req@ffff969e10b60a80 x1855007923560448/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075026.021405:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923560448/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075026.021420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075026.021427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075026.021432:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560448:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769075026.980785:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1769075027.295049:0:32693:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff969e09d26c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1769075027.295060:0:32693:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff969e09d26c08),name=/ 00000002:00010000:2.0:1769075027.295078:0:32693:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1769075027.295126:0:32693:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e09315800/0x44e2946041b61961 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 32693 initiator: MDT0 00010000:00010000:2.0:1769075027.295134:0:32693:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e09315800/0x44e2946041b61961 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 32693 initiator: MDT0 00010000:00010000:2.0:1769075027.295150:0:32693:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e09315800/0x44e2946041b61961 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 32693 initiator: MDT0 00000100:00080000:2.0:1769075027.295171:0:32693:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff969e0a46dc00 x1855007923560576/t0(0) o101->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1769075027.295179:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075028.004763:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075028.324720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075029.028740:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075029.348688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075030.052771:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075030.372693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075031.076736:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075031.076745:0:24457:0:(pinger.c:247:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1769075031.076748:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075031.076751:0:24457:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1769075031.076753:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075031.076756:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075031.076759:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075031.076767:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1720 00000100:00080000:2.0:1769075031.076778:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075031.076784:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075031.076795:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00010000:00010000:0.0:1769075031.076797:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075031.076829:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923560704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075031.076865:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075031.076870:0:24457:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1769075031.076893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560704:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769075031.076894:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075031.076901:0:24457:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1769075031.396674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075032.100760:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769075032.164999:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.204.143:988 00000400:00000100:2.0:1769075032.165055:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769075032.165073:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:0.0:1769075032.165202:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075031/real 1769075032] req@ffff969e0a46ed80 x1855007923560704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075032.165260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923560704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075032.165275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075032.165281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075032.165286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075032.420723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075033.124760:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075033.444727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075034.148764:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075034.468709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075035.172732:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075015/real 1769075015] req@ffff969e10b62a00 x1855007923559936/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1769075035.172749:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075035.172780:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923559936/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075035 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075035.172802:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075035.172823:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075035.172827:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923559936:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075035.172843:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075035.172874:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075035.172878:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075035.172886:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1710 00000100:00080000:3.0:1769075035.172899:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1769075035.172908:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075035.172949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923560832/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075035.172977:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075035.172982:0:32130:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1769075035.172985:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075035.172989:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075035.172992:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075035.172998:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1725 00000100:00080000:3.0:1769075035.173003:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075035.173009:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075035.173021:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075035.173023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923560832:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075035.173038:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923560960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075035.173051:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075035.173056:0:32130:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1769075035.173060:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075035.173064:0:32130:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000800:00000100:0.0:1769075035.173104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075035.173138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075035/real 1769075035] req@ffff969e0a7fca80 x1855007923560832/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075035.173149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923560832/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075035.173156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075035.173159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075035.173161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923560832:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075035.173179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560960:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075035.173198:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075035.173206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075035/real 1769075035] req@ffff969e0a7fe300 x1855007923560960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075035.173215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923560960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075035.173220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075035.173222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075035.173224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560960:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075035.492658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075036.196691:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075016/real 1769075016] req@ffff969e0a46e300 x1855007923560192/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075036.196728:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923560192/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075036 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075036.196747:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075016/real 1769075016] req@ffff969e0a46d880 x1855007923560064/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1769075036.196749:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075036.196766:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923560064/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075036 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075036.196792:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075036.196821:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075036.196833:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560192:192.168.204.143@tcp:8:kworker.0 00000100:00080000:0.0:1769075036.196841:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075036.196860:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075036.196868:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075036.196874:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075036.196884:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1729 00000100:00080000:0.0:1769075036.196885:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075036.196893:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923560064:192.168.204.143@tcp:8:kworker.0 00000100:00080000:3.0:1769075036.196901:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075036.196912:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075036.196953:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff480 x1855007923561088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075036.196986:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075036.196992:0:32130:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1769075036.196996:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075036.197000:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075036.197002:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075036.197008:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1729 00000100:00080000:3.0:1769075036.197012:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075036.197016:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075036.197020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923561088:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075036.197026:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075036.197041:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923561216/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075036.197050:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075036.197052:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075036.197054:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075036.197056:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1711 00000100:00080000:3.0:1769075036.197069:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1769075036.197072:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075036.197077:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075036.197086:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923561344/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075036.197094:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075036.197096:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075036.197098:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075036.197100:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1711 00000100:00080000:3.0:1769075036.197103:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1769075036.197105:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075036.197114:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075036.197125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075036.197125:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923561472/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075036.197147:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075036.197151:0:32130:0:(pinger.c:247:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1769075036.197160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561344:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075036.197179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075036.516696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075037.220747:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075037.540697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075038.244747:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075038.309034:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075036/real 1769075038] req@ffff969e0a7fdf80 x1855007923561472/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075038.309068:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923561472/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075038.309079:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075038.309085:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075038.309089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075038.309097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075036/real 1769075038] req@ffff969e0a7fce00 x1855007923561216/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075038.309110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923561216/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075038.309117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075038.309121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075038.309125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075038.309146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075036/real 1769075038] req@ffff969e0a7fed80 x1855007923561344/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075038.309158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923561344/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075038.309166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075038.309170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075038.309173:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561344:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075038.309180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075036/real 1769075038] req@ffff969e0a7ff480 x1855007923561088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075038.309191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff480 x1855007923561088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075038.309199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075038.309203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075038.309206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923561088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075038.564646:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00100000:0.0:1769075039.268721:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075014/real 1769075014] req@ffff969e0a46ce00 x1855007923559808/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075039.268762:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923559808/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075039 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075039.268784:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:3.0:1769075039.268791:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075039.268812:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075039.268819:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923559808:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075039.269015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923561600:192.168.204.143@tcp:250:kworker.0 00000800:00000100:0.0:1769075039.269091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075039.269129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075039/real 1769075039] req@ffff969e10b62680 x1855007923561600/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075039.269161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923561600/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075039.269180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075039.269188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075039.269194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923561600:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075039.269215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561728:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075039.269247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075039.269266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075039/real 1769075039] req@ffff969e10b62a00 x1855007923561728/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075039.269289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923561728/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075039.269315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075039.269320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075039.269325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561728:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075039.269333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075039.269356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075039.269367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075039/real 1769075039] req@ffff969e10b60a80 x1855007923561856/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075039.269387:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923561856/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075039.269394:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075039.269397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075039.269399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075039.269410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561984:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075039.269426:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075039.269433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075039/real 1769075039] req@ffff969e10b61f80 x1855007923561984/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075039.269441:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923561984/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075039.269459:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075039.269462:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075039.269464:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923561984:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075039.269471:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075039.269492:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075039.269503:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075039/real 1769075039] req@ffff969e10b61500 x1855007923562112/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075039.269516:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923562112/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075039.269537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075039.269542:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075039.269546:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075039.588722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075040.292814:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075040.612716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075041.316746:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075041.636679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075042.340761:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075042.660709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075043.364745:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075043.684689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075044.388696:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075044.388706:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075044.388710:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075044.388718:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1733 00000100:00080000:3.0:1769075044.388743:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075044.388749:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075044.388755:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075044.388780:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923562240/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075044.388802:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075044.388805:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075044.388806:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075044.388809:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1733 00000100:00080000:3.0:1769075044.388812:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075044.388815:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075044.388822:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075044.388831:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923562368/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075044.388835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923562240:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075044.388848:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075044.388850:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075044.388851:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075044.388854:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1733 00000100:00080000:3.0:1769075044.388856:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075044.388859:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075044.388862:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075044.388873:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923562496/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075044.388879:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075044.388886:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075044.388888:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075044.388891:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1733 00000100:00080000:3.0:1769075044.388895:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769075044.388900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075044.388903:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075044.388911:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075044.388926:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923562624/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075044.388931:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075044/real 1769075044] req@ffff969e0a7fc380 x1855007923562240/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075044.388957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923562240/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075044.388962:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075044.388966:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075044.388966:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075044.388971:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075044.388975:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075044.388979:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1733 00000100:00100000:0.0:1769075044.388982:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923562240:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075044.388985:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075044.388991:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075044.388995:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075044.389004:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562368:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075044.389009:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923562752/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769075044.389027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075044.389038:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075044/real 1769075044] req@ffff969e0a7ffb80 x1855007923562368/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075044.389047:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923562368/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075044.389063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075044.389066:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075044.389068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562368:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075044.389073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075044.389086:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075044.389093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075044/real 1769075044] req@ffff969e0a7fc000 x1855007923562496/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075044.389100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923562496/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075044.389105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075044.389106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075044.389108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075044.389112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562624:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075044.389133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075044.389140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075044/real 1769075044] req@ffff969e0a7ff800 x1855007923562624/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075044.389147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923562624/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075044.389152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075044.389158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075044.389165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562624:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075044.389178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075044.389191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075044.389198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075044/real 1769075044] req@ffff969e0a7fd500 x1855007923562752/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075044.389205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923562752/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075044.389211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075044.389213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075044.389215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923562752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075044.708657:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075045.412768:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075045.732663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075046.436798:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075046.756705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075047.460723:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075047.780708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075048.484733:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075048.804681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075049.508656:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075049.508665:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075049.508669:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075049.508678:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1738 00000100:00080000:3.0:1769075049.508690:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075049.508718:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075049.508736:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075049.508753:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923562880/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075049.508780:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075049.508784:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075049.508786:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075049.508790:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1738 00000100:00080000:3.0:1769075049.508795:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075049.508799:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075049.508809:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923562880:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075049.508809:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075049.508823:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923563008/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075049.508833:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075049.508835:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075049.508837:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075049.508840:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1738 00000100:00080000:3.0:1769075049.508844:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075049.508847:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075049.508851:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075049.508859:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923563136/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075049.508869:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075049.508871:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075049.508881:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075049.508886:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1738 00000100:00080000:3.0:1769075049.508890:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075049.508894:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075049.508907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563008:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075049.508907:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075049.508919:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923563264/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075049.508929:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075049.508933:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075049.508935:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075049.508940:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1738 00000100:00080000:3.0:1769075049.508943:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075049.508947:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563136:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075049.508947:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075049.508953:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075049.508963:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923563392/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075049.508976:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563264:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075049.509006:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075049.828685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075050.532760:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075050.597052:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075049/real 1769075050] req@ffff969e0a7fc000 x1855007923563264/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075050.597106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923563264/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075050.597119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075050.597124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075050.597128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563264:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075050.597136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075049/real 1769075050] req@ffff969e0a7ffb80 x1855007923563392/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075050.597150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923563392/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075050.597158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075050.597162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075050.597165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075050.597171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075049/real 1769075050] req@ffff969e0a7fd500 x1855007923563008/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075050.597180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923563008/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075050.597188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075050.597191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075050.597195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563008:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075050.597201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075049/real 1769075050] req@ffff969e0a7ff800 x1855007923563136/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075050.597224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923563136/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075050.597232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075050.597236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075050.597240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075050.597246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075049/real 1769075050] req@ffff969e0a7fdc00 x1855007923562880/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075050.597298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923562880/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075050.597305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075050.597309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075050.597312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923562880:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075050.852713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075051.556762:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075051.876676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075052.580804:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075052.900727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075053.604774:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075053.924691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075054.628695:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075054.628709:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075054.628715:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075054.628729:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1743 00000100:00080000:3.0:1769075054.628744:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075054.628753:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075054.628772:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075054.628802:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923563520/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075054.628856:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075054.628861:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075054.628863:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075054.628869:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1743 00000100:00080000:3.0:1769075054.628875:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075054.628880:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075054.628886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923563520:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075054.628894:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075054.628914:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff480 x1855007923563648/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075054.628923:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075054.628926:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075054.628927:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075054.628930:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1743 00000100:00080000:3.0:1769075054.628933:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075054.628936:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075054.628940:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075054.628948:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923563776/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075054.628954:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075054.628957:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075054.628959:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075054.628961:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1743 00000100:00080000:3.0:1769075054.628971:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075054.628974:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:0.0:1769075054.628980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075054.628981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075054.628992:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923563904/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075054.628999:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075054.629002:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075054.629004:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075054.629007:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1743 00000100:00080000:3.0:1769075054.629010:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075054.629012:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075054.629015:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075054.629022:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923564032/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075054.629028:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075054/real 1769075054] req@ffff969e0a7fc380 x1855007923563520/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075054.629048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923563520/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075054.629069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075054.629073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075054.629076:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923563520:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075054.629099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563648:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075054.629118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075054.629127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075054/real 1769075054] req@ffff969e0a7ff480 x1855007923563648/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075054.629135:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff480 x1855007923563648/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075054.629141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075054.629143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075054.629145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563648:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075054.629149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075054.629159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075054.629178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075054/real 1769075054] req@ffff969e0a7fed80 x1855007923563776/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075054.629205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923563776/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075054.629214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075054.629218:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075054.629221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075054.629236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563904:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075054.629266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075054.629275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075054/real 1769075054] req@ffff969e0a7fce00 x1855007923563904/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075054.629283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923563904/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075054.629289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075054.629292:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075054.629294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923563904:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075054.629298:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075054.629309:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075054.629314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075054/real 1769075054] req@ffff969e0a7fdf80 x1855007923564032/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075054.629322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923564032/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075054.629333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075054.629335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075054.629337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075054.948678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075055.652786:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075055.972697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075056.676777:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075056.996673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075057.700734:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075058.020655:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075058.724799:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075059.044724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075059.748651:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075059.748668:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075059.748673:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075059.748688:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1748 00000100:00080000:3.0:1769075059.748705:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075059.748717:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075059.748771:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923564160/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075059.748791:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075059.748809:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075059.748840:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075059.748842:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075059.748848:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1748 00000100:00080000:3.0:1769075059.748854:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075059.748862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923564160:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075059.748862:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075059.748876:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075059.748892:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923564288/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075059.748902:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075059.748904:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075059.748906:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075059.748909:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1748 00000100:00080000:3.0:1769075059.748931:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075059.748934:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075059.748937:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075059.748945:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075059.748950:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923564416/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075059.748963:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075059.748968:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075059.748971:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075059.748976:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1748 00000100:00080000:3.0:1769075059.748981:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075059.748987:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075059.748995:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075059/real 1769075059] req@ffff969e0a7fe300 x1855007923564160/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075059.749004:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075059.749025:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923564544/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075059.749029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923564160/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075059.749049:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075059.749053:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075059.749053:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075059.749057:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075059.749062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075059.749065:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1748 00000100:00100000:2.0:1769075059.749070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923564160:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075059.749072:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075059.749080:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075059.749084:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075059.749095:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923564672/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075059.749124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564288:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075059.749154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075059.749169:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075059/real 1769075059] req@ffff969e0a7fca80 x1855007923564288/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075059.749183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923564288/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075059.749190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075059.749192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075059.749201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564288:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075059.749207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075059.749228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075059.749241:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075059/real 1769075059] req@ffff969e0a7fe680 x1855007923564416/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075059.749254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923564416/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075059.749263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075059.749267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075059.749271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075059.749277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564544:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075059.749294:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075059.749301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075059/real 1769075059] req@ffff969e0a7fc700 x1855007923564544/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075059.749310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923564544/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075059.749315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075059.749318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075059.749320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564544:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075059.749324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075059.749334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075059.749347:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075059/real 1769075059] req@ffff969e0a7fd180 x1855007923564672/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075059.749355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923564672/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075059.749361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075059.749362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075059.749364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075060.068659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075060.772756:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075061.092746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075061.796890:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075062.116716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075062.820797:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075063.140674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075063.844799:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075064.164737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075064.868678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075064.868688:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075064.868692:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075064.868700:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1754 00000100:00080000:3.0:1769075064.868709:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075064.868715:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075064.868743:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075064.868748:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923564800/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075064.868771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075064.868778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075064.868780:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075064.868783:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1754 00000100:00080000:3.0:1769075064.868800:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075064.868804:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075064.868812:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075064.868816:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923564800:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075064.868824:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923564928/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075064.868833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075064.868836:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075064.868837:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075064.868840:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1754 00000100:00080000:3.0:1769075064.868843:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075064.868845:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075064.868848:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075064.868856:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923565056/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075064.868863:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075064.868866:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075064.868867:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075064.868869:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1754 00000100:00080000:3.0:1769075064.868871:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075064.868875:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075064.868878:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075064.868883:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075064.868895:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923565184/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075064.868911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075064.868913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075064.868915:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075064.868919:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1754 00000100:00080000:3.0:1769075064.868924:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075064.868928:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075064/real 1769075064] req@ffff969e0a7fd880 x1855007923564800/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075064.868930:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075064.868937:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075064.868949:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923565312/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075064.868951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923564800/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075064.868975:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075064.868979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075064.868983:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923564800:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075064.869002:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564928:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075064.869036:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075064.869053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075064/real 1769075064] req@ffff969e0a7fea00 x1855007923564928/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075064.869062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923564928/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075064.869068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075064.869070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075064.869073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923564928:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075064.869077:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075064.869090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075064.869096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075064/real 1769075064] req@ffff969e0a7fd180 x1855007923565056/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075064.869103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923565056/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075064.869108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075064.869109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075064.869111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075064.869115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565184:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075064.869125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075064.869130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075064/real 1769075064] req@ffff969e0a7fc700 x1855007923565184/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075064.869137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923565184/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075064.869147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075064.869150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075064.869152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565184:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075064.869156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075064.869168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075064.869173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075064/real 1769075064] req@ffff969e0a7fe680 x1855007923565312/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075064.869180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923565312/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075064.869185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075064.869187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075064.869189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075065.188671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075065.892817:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075066.212697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075066.916746:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075067.236725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075067.940748:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075068.260687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075068.964756:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075069.284683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075069.988655:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075069.988666:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075069.988673:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075069.988684:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1759 00000100:00080000:3.0:1769075069.988694:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075069.988701:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075069.988732:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923565440/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075069.988757:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075069.988760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075069.988762:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769075069.988765:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075069.988766:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1759 00000100:00080000:3.0:1769075069.988770:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075069.988773:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075069.988796:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075069.988809:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923565568/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075069.988810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923565440:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075069.988826:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075069.988828:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075069.988830:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075069.988833:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1759 00000100:00080000:3.0:1769075069.988837:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075069.988840:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075069.988844:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075069.988852:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923565696/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075069.988858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075069.988861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075069.988862:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075069.988865:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1759 00000100:00080000:3.0:1769075069.988868:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075069.988870:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075069.988873:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075069.988880:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923565824/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075069.988886:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075069.988889:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075069.988895:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075069.988899:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1759 00000100:00080000:3.0:1769075069.988902:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075069.988907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565568:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075069.988908:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075069.988915:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075069.988944:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923565952/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075069.988966:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075069.988997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565824:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075069.989036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075070.308657:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075071.012719:0:15141:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075071.332654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075072.036702:0:15142:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769075072.101039:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.204.143:988 00000400:00000100:0.0:1769075072.101104:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769075072.101126:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075072.101178:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075072.101189:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075072.101201:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075072.101211:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075072.101249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075069/real 1769075072] req@ffff969e0a7fed80 x1855007923565952/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075072.101279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923565952/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075072.101290:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075072.101295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075072.101300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075072.101308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075069/real 1769075072] req@ffff969e0a7fe300 x1855007923565568/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075072.101329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923565568/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075072.101337:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075072.101342:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075072.101345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565568:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075072.101353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075069/real 1769075072] req@ffff969e0a7fdf80 x1855007923565696/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075072.101366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923565696/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075072.101400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075072.101405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075072.101409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075072.101417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075069/real 1769075072] req@ffff969e0a7fce00 x1855007923565824/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075072.101430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923565824/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075072.101438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075072.101441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075072.101445:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923565824:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075072.101453:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075069/real 1769075072] req@ffff969e0a7fca80 x1855007923565440/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075072.101465:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923565440/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075072.101473:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075072.101476:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075072.101479:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923565440:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075072.356677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075073.060730:0:15141:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 16 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1769075073.060740:0:15141:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e05655000/0x44e2946041b60907 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b91fc930f88b2e9 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1769075073.060793:0:15141:0:(osc_cache.c:1892:try_to_add_extent_for_io()) ### extent: 00000000e7c05bc1 ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e05655000/0x44e2946041b60907 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88b2e9 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1769075073.060810:0:15141:0:(osc_cache.c:1103:osc_extent_make_ready()) ### extent: 00000000e7c05bc1 ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e05655000/0x44e2946041b60907 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88b2e9 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075073.060869:0:15142:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 16 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1769075073.060939:0:15141:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e05655000/0x44e2946041b60907 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88b2e9 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1769075073.061072:0:15141:0:(osc_request.c:1967:osc_brw_prep_request()) brw rpc ffff969e06a61c00 - object 0x280000401:460 offset 0<>7 00000100:00100000:0.0:1769075073.061147:0:2418:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:0.0:1769075073.061170:0:2418:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff969e06a61c00 x1855007923566080/t0(0) o4->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:6/4 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1769075073.061392:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075073.061395:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075073.061397:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075073.061434:0:15142:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e05654800/0x44e2946041b6091c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b91fc930f88b321 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1769075073.061451:0:15142:0:(osc_cache.c:1892:try_to_add_extent_for_io()) ### extent: 00000000e198f9f8 ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e05654800/0x44e2946041b6091c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88b321 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1769075073.061459:0:15142:0:(osc_cache.c:1103:osc_extent_make_ready()) ### extent: 00000000e198f9f8 ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e05654800/0x44e2946041b6091c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88b321 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075073.061481:0:15142:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e05654800/0x44e2946041b6091c lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88b321 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1769075073.061511:0:15142:0:(osc_request.c:1967:osc_brw_prep_request()) brw rpc ffff969e0a7ff480 - object 0x2c0000401:460 offset 0<>7 00000100:00100000:3.0:1769075073.061541:0:2420:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1769075073.061552:0:2420:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff969e0a7ff480 x1855007923566208/t0(0) o4->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:6/4 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1769075073.380708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075074.084704:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075074.404651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075075.108666:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075075.108675:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075075.108695:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075075.108703:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1764 00000100:00080000:3.0:1769075075.108729:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075075.108737:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075075.108748:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075075.108771:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923566336/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075075.108819:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075075.108823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075075.108825:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075075.108829:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1764 00000100:00080000:3.0:1769075075.108833:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075075.108838:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075075.108858:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075075.108870:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923566464/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075075.108879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923566336:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075075.108880:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075075.108887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075075.108889:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075075.108892:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1764 00000100:00080000:3.0:1769075075.108895:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075075.108901:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075075.108906:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075075.108914:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923566592/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075075.108934:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075075.108937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075075.108938:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075075.108941:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1764 00000100:00080000:3.0:1769075075.108952:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075075.108954:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075075.108958:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075075.108964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075075.108970:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923566720/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075075.108978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075075.108980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075075.108982:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075075.108984:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1764 00000100:00080000:3.0:1769075075.108987:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075075.108990:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075075.108993:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075075.109001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075075/real 1769075075] req@ffff969e0a7fc380 x1855007923566336/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075075.109003:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923566848/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075075.109042:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923566336/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075075.109054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075075.109059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075075.109063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923566336:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075075.109085:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923566464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075075.109118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075075.109130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075075/real 1769075075] req@ffff969e0a7fdc00 x1855007923566464/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075075.109143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923566464/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075075.109151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075075.109154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075075.109158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923566464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075075.109163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923566592:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075075.109180:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075075.109190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075075/real 1769075075] req@ffff969e0a7ff800 x1855007923566592/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075075.109216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923566592/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075075.109224:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075075.109227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075075.109230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923566592:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075075.109236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923566720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075075.109258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075075.109270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075075/real 1769075075] req@ffff969e0a7fd500 x1855007923566720/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075075.109280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923566720/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075075.109286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075075.109288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075075.109290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923566720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075075.109294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923566848:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075075.109307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075075.109313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075075/real 1769075075] req@ffff969e0a7ffb80 x1855007923566848/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075075.109320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923566848/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075075.109334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075075.109337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075075.109339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923566848:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075075.428658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075076.132845:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075076.452730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075077.156718:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075077.476689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075078.180714:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075078.500726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075079.204812:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075079.524802:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075080.228675:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075080.228686:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075080.228691:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075080.228701:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1769 00000100:00080000:3.0:1769075080.228729:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075080.228737:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075080.228766:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075080.228791:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923566976/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075080.228816:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075080.228819:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075080.228820:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075080.228824:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1769 00000100:00080000:3.0:1769075080.228827:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075080.228830:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075080.228841:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075080.228854:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923567104/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075080.228857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923566976:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075080.228870:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075080.228874:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075080.228876:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075080.228882:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1769 00000100:00080000:3.0:1769075080.228886:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075080.228890:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075080.228896:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075080.228909:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923567232/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075080.228920:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075080.228932:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075080.228934:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075080.228937:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1769 00000800:00000100:2.0:1769075080.228943:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075080.228943:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075080.228951:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075080.228955:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075080.228966:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923567360/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075080.228974:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075080.228977:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075080.228979:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075080.228983:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1769 00000100:00100000:2.0:1769075080.228984:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075080/real 1769075080] req@ffff969e0a7fc000 x1855007923566976/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075080.228988:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075080.228994:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075080.229000:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075080.229012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923566976/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075080.229012:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923567488/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075080.229040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075080.229047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075080.229053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923566976:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075080.229082:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075080.229108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075080.229122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075080/real 1769075080] req@ffff969e0a7ffb80 x1855007923567104/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075080.229132:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923567104/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075080.229138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075080.229140:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075080.229149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075080.229155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567232:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075080.229170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075080.229178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075080/real 1769075080] req@ffff969e0a7fd500 x1855007923567232/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075080.229186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923567232/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075080.229191:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075080.229193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075080.229195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567232:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075080.229199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075080.229210:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075080.229219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075080/real 1769075080] req@ffff969e0a7ff800 x1855007923567360/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075080.229231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923567360/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075080.229240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075080.229244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075080.229247:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075080.229254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567488:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075080.229276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075080.229290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075080/real 1769075080] req@ffff969e0a7fdc00 x1855007923567488/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075080.229298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923567488/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075080.229305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075080.229307:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075080.229309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567488:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075080.548821:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075081.252797:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075081.572705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075082.276832:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075082.596675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075083.300764:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075083.620693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075084.324767:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075084.644676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075085.348690:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075085.348702:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075085.348707:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075085.348718:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1774 00000100:00080000:3.0:1769075085.348730:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075085.348745:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075085.348750:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075085.348779:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923567616/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075085.348814:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075085.348816:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075085.348818:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075085.348821:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1774 00000100:00080000:3.0:1769075085.348824:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075085.348828:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075085.348835:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075085.348846:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923567744/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075085.348855:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075085.348859:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769075085.348861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923567616:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075085.348861:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075085.348866:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1774 00000100:00080000:3.0:1769075085.348870:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075085.348874:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075085.348877:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075085.348885:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923567872/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075085.348892:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075085.348894:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075085.348896:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075085.348898:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1774 00000100:00080000:3.0:1769075085.348933:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075085.348936:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075085.348940:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075085.348943:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075085.348952:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923568000/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075085.348960:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075085.348963:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075085.348964:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075085.348967:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1774 00000100:00080000:3.0:1769075085.348970:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075085.348973:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075085.348976:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075085.348978:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075085/real 1769075085] req@ffff969e0a7fc380 x1855007923567616/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075085.348987:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923568128/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075085.348999:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923567616/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075085.349009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075085.349013:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075085.349016:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923567616:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075085.349042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075085.349059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075085.349068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075085/real 1769075085] req@ffff969e0a7fca80 x1855007923567744/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075085.349075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923567744/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075085.349092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075085.349095:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075085.349097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075085.349101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567872:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075085.349115:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075085.349121:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075085/real 1769075085] req@ffff969e0a7fce00 x1855007923567872/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075085.349129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923567872/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075085.349134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075085.349135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075085.349137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923567872:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075085.349141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075085.349160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075085.349166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075085/real 1769075085] req@ffff969e0a7fdf80 x1855007923568000/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075085.349174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923568000/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075085.349179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075085.349181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075085.349190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075085.349197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568128:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075085.349219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075085.349231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075085/real 1769075085] req@ffff969e0a7fe300 x1855007923568128/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075085.349242:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923568128/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075085.349252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075085.349255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075085.349259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568128:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075085.668656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075086.372768:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075086.692720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075087.396811:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075087.716697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075088.420740:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075088.740712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075089.444827:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075089.764768:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075090.468716:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075090.468733:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075090.468738:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075090.468753:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1779 00000100:00080000:3.0:1769075090.468768:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075090.468780:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075090.468795:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075090.468830:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923568256/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075090.468864:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075090.468889:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075090.468892:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075090.468899:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1779 00000100:00080000:3.0:1769075090.468904:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075090.468912:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075090.468917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923568256:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075090.468928:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075090.468953:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923568384/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075090.468968:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075090.468973:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075090.468975:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075090.468980:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1779 00000100:00080000:3.0:1769075090.468989:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075090.468994:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075090.469002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075090.469010:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075090.469030:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923568512/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075090.469040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075090/real 1769075090] req@ffff969e0a7fed80 x1855007923568256/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075090.469047:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075090.469057:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075090.469059:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075090.469064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923568256/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075090.469071:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1779 00000100:00080000:3.0:1769075090.469081:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075090.469082:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075090.469100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075090.469100:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075090.469106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923568256:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075090.469119:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075090.469129:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075090.469140:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923568640/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075090.469152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075090.469161:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075090.469167:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075090.469171:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075090.469178:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1779 00000100:00100000:2.0:1769075090.469185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075090/real 1769075090] req@ffff969e0a7fe680 x1855007923568384/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075090.469190:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075090.469204:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075090.469206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923568384/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075090.469221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075090.469221:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075090.469230:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075090.469234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075090.469239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568512:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075090.469248:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923568768/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075090.469260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075090.469270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075090/real 1769075090] req@ffff969e0a7fc700 x1855007923568512/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075090.469279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923568512/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075090.469287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075090.469290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075090.469292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568512:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075090.469304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075090.469315:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075090.469321:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075090/real 1769075090] req@ffff969e0a7fd180 x1855007923568640/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075090.469328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923568640/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075090.469338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075090.469341:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075090.469343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075090.469347:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568768:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075090.469360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075090.469367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075090/real 1769075090] req@ffff969e0a7fea00 x1855007923568768/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075090.469379:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923568768/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075090.469388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075090.469392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075090.469395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923568768:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075090.788715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075091.492764:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075091.812690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075092.516717:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075092.836749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075093.540805:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075093.860683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075094.564751:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075094.884698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075095.588681:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075095.588691:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075095.588695:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075095.588703:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1784 00000100:00080000:3.0:1769075095.588714:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075095.588720:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075095.588742:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075095.588756:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923568896/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075095.588779:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075095.588781:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075095.588782:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075095.588785:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1784 00000100:00080000:3.0:1769075095.588817:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075095.588822:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075095.588830:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075095.588844:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923569024/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075095.588854:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075095.588859:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923568896:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075095.588860:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075095.588871:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075095.588879:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1784 00000100:00080000:3.0:1769075095.588883:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075095.588890:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075095.588896:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075095.588911:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923569152/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075095.588922:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075095.588926:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075095.588928:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075095.588932:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1784 00000100:00080000:3.0:1769075095.588940:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075095.588944:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075095.588948:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075095.588953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075095.588964:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923569280/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075095.588984:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075095.588987:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075095.588989:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075095.588996:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075095/real 1769075095] req@ffff969e0a7fd880 x1855007923568896/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075095.589001:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1784 00000100:00080000:3.0:1769075095.589012:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075095.589025:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075095.589031:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923568896/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075095.589031:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075095.589049:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923569408/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075095.589051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075095.589067:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075095.589076:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923568896:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075095.589117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075095.589136:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075095.589144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075095/real 1769075095] req@ffff969e0a7fea00 x1855007923569024/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075095.589153:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923569024/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075095.589159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075095.589161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075095.589164:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075095.589168:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569152:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075095.589179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075095.589185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075095/real 1769075095] req@ffff969e0a7fd180 x1855007923569152/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075095.589192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923569152/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075095.589197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075095.589199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075095.589201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569152:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075095.589205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075095.589215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075095.589220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075095/real 1769075095] req@ffff969e0a7fc700 x1855007923569280/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075095.589234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923569280/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075095.589241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075095.589243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075095.589245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075095.589249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569408:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075095.589261:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075095.589267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075095/real 1769075095] req@ffff969e0a7fe680 x1855007923569408/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075095.589274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923569408/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075095.589279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075095.589281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075095.589283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569408:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075095.908673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075096.612762:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075096.932672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075097.636742:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075097.956729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075098.660774:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075098.980710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075099.684702:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075100.004705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075100.708650:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075100.708662:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075100.708667:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075100.708682:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1789 00000100:00080000:3.0:1769075100.708695:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075100.708704:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075100.708728:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075100.708762:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923569536/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075100.708795:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075100.708798:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075100.708805:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075100.708809:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1789 00000100:00080000:3.0:1769075100.708813:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075100.708817:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075100.708834:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075100.708850:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923569664/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075100.708851:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923569536:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075100.708867:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075100.708870:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075100.708872:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075100.708875:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1789 00000100:00080000:3.0:1769075100.708888:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075100.708891:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075100.708894:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075100.708903:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923569792/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075100.708911:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075100.708914:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075100.708915:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075100.708918:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1789 00000100:00080000:3.0:1769075100.708938:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075100.708940:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075100.708944:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075100.708954:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075100.708955:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923569920/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075100.708964:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075100.708967:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075100.708973:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075100.708978:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1789 00000100:00080000:3.0:1769075100.708982:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075100.708996:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075100.709006:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075100.709007:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075100/real 1769075100] req@ffff969e0a7fed80 x1855007923569536/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075100.709020:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923570048/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075100.709029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923569536/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075100.709053:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075100.709057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075100.709061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923569536:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075100.709084:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075100.709103:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075100.709112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075100/real 1769075100] req@ffff969e0a7fe300 x1855007923569664/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075100.709121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923569664/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075100.709127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075100.709129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075100.709132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075100.709135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569792:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075100.709146:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075100.709151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075100/real 1769075100] req@ffff969e0a7fdf80 x1855007923569792/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075100.709159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923569792/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075100.709164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075100.709166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075100.709168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569792:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075100.709178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075100.709193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075100.709199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075100/real 1769075100] req@ffff969e0a7fce00 x1855007923569920/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075100.709207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923569920/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075100.709213:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075100.709215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075100.709217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923569920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075100.709221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570048:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075100.709231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075100.709236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075100/real 1769075100] req@ffff969e0a7fca80 x1855007923570048/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075100.709244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923570048/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075100.709249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075100.709251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075100.709253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570048:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075101.028710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075101.732763:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075102.052703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075102.756752:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075103.076729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075103.780784:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075104.100712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075104.804795:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075105.124693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075105.828709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075105.828719:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075105.828723:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075105.828749:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1795 00000100:00080000:2.0:1769075105.828758:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075105.828765:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075105.828785:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075105.828797:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923570176/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075105.828821:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075105.828823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075105.828825:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075105.828827:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1795 00000100:00080000:2.0:1769075105.828830:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075105.828833:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075105.828841:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075105.828852:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923570304/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075105.828860:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923570176:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075105.828867:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075105.828873:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075105.828875:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075105.828880:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1795 00000100:00080000:2.0:1769075105.828885:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075105.828890:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075105.828897:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075105.828912:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923570432/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075105.828938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075105.828941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075105.828944:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075105.828949:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1795 00000100:00080000:2.0:1769075105.828961:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075105.828967:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075105.828969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570304:192.168.204.143@tcp:38:kworker.0 00000008:00100000:2.0:1769075105.828973:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075105.828988:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923570560/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075105.828998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075105.829002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075105.829004:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075105.829007:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570432:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769075105.829012:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1795 00000100:00080000:2.0:1769075105.829021:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075105.829026:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075105.829032:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075105.829042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075105.829049:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923570688/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075105.829079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570688:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075105.893176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075105/real 1769075105] req@ffff969e0a46ed80 x1855007923570688/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075105.893202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923570688/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075105.893215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075105.893221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075105.893225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570688:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075105.893233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075105/real 1769075105] req@ffff969e0a46e300 x1855007923570560/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075105.893270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923570560/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075105.893279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075105.893282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075105.893286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075105.893293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075105/real 1769075105] req@ffff969e0a46ce00 x1855007923570304/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075105.893306:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923570304/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075105.893314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075105.893317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075105.893321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075105.893359:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075105/real 1769075105] req@ffff969e0a46d880 x1855007923570432/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075105.893372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923570432/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075105.893379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075105.893383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075105.893386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570432:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075105.893394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075105/real 1769075105] req@ffff969e0a46ca80 x1855007923570176/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075105.893415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923570176/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075105.893423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075105.893427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075105.893431:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923570176:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075106.148766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075106.852804:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075107.172761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075107.876788:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075108.196705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075108.900800:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075109.220726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075109.924792:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075110.244707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075110.948689:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075110.948702:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075110.948707:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075110.948718:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1800 00000100:00080000:2.0:1769075110.948757:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075110.948776:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075110.948806:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075110.948827:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923570816/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075110.948853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075110.948856:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075110.948857:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075110.948861:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1800 00000100:00080000:2.0:1769075110.948863:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075110.948866:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075110.948876:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075110.948886:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923570944/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075110.948895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075110.948896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923570816:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075110.948899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075110.948903:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075110.948908:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1800 00000100:00080000:2.0:1769075110.948911:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075110.948913:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075110.948916:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075110.948925:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923571072/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075110.948932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075110.948940:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075110.948942:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075110.948945:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1800 00000100:00080000:2.0:1769075110.948949:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075110.948952:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075110.948956:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075110.948966:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923571200/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075110.948973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075110.948976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075110.948979:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769075110.948981:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075110.948985:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1800 00000100:00080000:2.0:1769075110.948989:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075110.948991:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075110.948995:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075110.949004:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923571328/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075110.949026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075110/real 1769075110] req@ffff969e0a46ea00 x1855007923570816/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075110.949043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923570816/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075110.949052:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075110.949056:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075110.949059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923570816:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075110.949093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075110.949110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075110.949119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075110/real 1769075110] req@ffff969e0a46df80 x1855007923570944/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075110.949127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923570944/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075110.949133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075110.949136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075110.949138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923570944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075110.949142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571072:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075110.949161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075110.949166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075110/real 1769075110] req@ffff969e0a46f480 x1855007923571072/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075110.949174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923571072/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075110.949179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075110.949181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075110.949183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571072:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075110.949186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075110.949197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075110.949202:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075110/real 1769075110] req@ffff969e0a46d500 x1855007923571200/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075110.949216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923571200/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075110.949223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075110.949225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075110.949227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075110.949231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571328:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075110.949269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075110.949286:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075110/real 1769075110] req@ffff969e0a46f800 x1855007923571328/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075110.949298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923571328/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075110.949308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075110.949311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075110.949315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571328:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075111.268683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075111.972765:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075112.292712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075112.996737:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075113.316711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075114.020817:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075114.340679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075115.044723:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075115.364688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075116.068684:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075116.068692:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075116.068696:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075116.068703:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1805 00000100:00080000:3.0:1769075116.068714:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075116.068721:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075116.068736:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075116.068753:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923571456/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075116.068776:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075116.068778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075116.068780:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075116.068783:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1805 00000100:00080000:3.0:1769075116.068786:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075116.068789:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075116.068797:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075116.068805:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923571456:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075116.068807:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923571584/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075116.068837:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075116.068839:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075116.068841:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075116.068844:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1805 00000100:00080000:3.0:1769075116.068847:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075116.068850:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075116.068853:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769075116.068867:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075116.068867:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923571712/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075116.068880:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075116.068885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075116.068887:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075116.068892:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1805 00000100:00080000:3.0:1769075116.068897:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769075116.068900:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075116/real 1769075116] req@ffff969e0a7fc380 x1855007923571456/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075116.068904:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075116.068915:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075116.068923:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923571456/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075116.068929:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923571840/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075116.068936:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075116.068939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075116.068942:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075116.068945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075116.068945:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075116.068952:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1805 00000100:00100000:0.0:1769075116.068956:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923571456:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075116.068961:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075116.068967:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075116.068971:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075116.068979:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923571968/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075116.068986:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075116.069012:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075116.069021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075116/real 1769075116] req@ffff969e0a7fdc00 x1855007923571584/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075116.069030:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923571584/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075116.069038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075116.069042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075116.069046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075116.069052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571712:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075116.069071:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075116.069082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075116/real 1769075116] req@ffff969e0a7ff800 x1855007923571712/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075116.069101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923571712/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075116.069107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075116.069109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075116.069111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571712:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075116.069116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075116.069131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075116.069138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075116/real 1769075116] req@ffff969e0a7fd500 x1855007923571840/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075116.069145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923571840/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075116.069150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075116.069168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075116.069171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075116.069184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571968:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075116.069204:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075116.069215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075116/real 1769075116] req@ffff969e0a7ffb80 x1855007923571968/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075116.069227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923571968/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075116.069259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075116.069261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075116.069264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923571968:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075116.388696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075117.092751:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075117.412727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075118.116791:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075118.436685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075119.140765:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075119.460755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075120.164844:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075120.484717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075121.188692:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075121.188701:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075121.188705:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075121.188713:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1810 00010000:00010000:2.0:1769075121.188717:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075121.188722:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075121.188729:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075121.188762:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923572096/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075121.188785:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075121.188788:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075121.188789:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075121.188792:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1810 00000100:00080000:0.0:1769075121.188794:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075121.188812:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075121.188820:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075121.188827:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923572096:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075121.188832:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923572224/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075121.188843:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075121.188846:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075121.188847:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075121.188849:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1810 00000100:00080000:0.0:1769075121.188854:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075121.188856:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075121.188859:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075121.188867:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923572352/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075121.188874:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075121.188877:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075121.188878:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075121.188880:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1810 00000100:00080000:0.0:1769075121.188882:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075121.188884:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075121.188887:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075121.188895:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923572480/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075121.188901:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075121.188903:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075121.188910:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075121.188912:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075121.188915:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1810 00000100:00080000:0.0:1769075121.188919:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075121.188922:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075121.188926:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075121.188934:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923572608/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075121.188945:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075121/real 1769075121] req@ffff969e06a61880 x1855007923572096/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075121.188967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923572096/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075121.188990:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075121.188995:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075121.188999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923572096:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075121.189024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075121.189051:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075121.189066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075121/real 1769075121] req@ffff969e06a63100 x1855007923572224/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075121.189116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923572224/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075121.189125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075121.189129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075121.189132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075121.189139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572352:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075121.189160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075121.189170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075121/real 1769075121] req@ffff969e06a62680 x1855007923572352/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075121.189178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923572352/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075121.189183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075121.189186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075121.189188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572352:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075121.189198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075121.189213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075121.189219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075121/real 1769075121] req@ffff969e06a61180 x1855007923572480/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075121.189227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923572480/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075121.189233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075121.189234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075121.189236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075121.189240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572608:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075121.189250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075121.189255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075121/real 1769075121] req@ffff969e06a61500 x1855007923572608/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075121.189262:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923572608/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075121.189267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075121.189269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075121.189271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572608:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075121.508704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075122.212756:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075122.532688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075123.236729:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075123.556722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075124.260861:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075125.284772:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075125.604846:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075126.308730:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075126.308741:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075126.308745:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075126.308755:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1815 00000100:00080000:0.0:1769075126.308784:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075126.308792:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075126.308834:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923572736/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075126.308834:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075126.308875:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075126.308879:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075126.308881:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075126.308886:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1815 00000100:00080000:0.0:1769075126.308890:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075126.308893:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075126.308909:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923572736:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769075126.308935:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075126.308956:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923572864/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075126.308972:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075126.309023:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075126.309026:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769075126.309031:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075126.309033:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1815 00000100:00080000:0.0:1769075126.309041:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075126.309047:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075126.309053:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075126.309067:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62a00 x1855007923572992/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075126.309075:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075126/real 1769075126] req@ffff969e06a62d80 x1855007923572736/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075126.309091:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075126.309100:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075126.309104:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075126.309109:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1815 00000100:00080000:0.0:1769075126.309113:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075126.309115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923572736/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075126.309120:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769075126.309130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769075126.309131:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075126.309136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075126.309141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923572736:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075126.309147:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923573120/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075126.309157:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075126.309160:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769075126.309163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572864:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075126.309171:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075126.309177:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1815 00000100:00080000:0.0:1769075126.309183:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075126.309185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075126.309192:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075126.309198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075126/real 1769075126] req@ffff969e06a60380 x1855007923572864/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075126.309199:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075126.309214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923572864/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075126.309221:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923573248/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075126.309223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075126.309231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075126.309239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075126.309250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572992:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075126.309274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075126.309305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075126/real 1769075126] req@ffff969e06a62a00 x1855007923572992/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075126.309314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62a00 x1855007923572992/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075126.309320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075126.309322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075126.309324:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923572992:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075126.309329:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075126.309343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075126.309349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075126/real 1769075126] req@ffff969e06a61f80 x1855007923573120/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075126.309356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923573120/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075126.309361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075126.309363:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075126.309365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075126.309376:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573248:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075126.309397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075126.309402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075126/real 1769075126] req@ffff969e06a60a80 x1855007923573248/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075126.309410:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923573248/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075126.309420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075126.309422:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075126.309424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573248:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075126.629078:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075127.332780:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075127.652700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075128.356840:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075128.676707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075129.380813:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075129.700687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075130.404896:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075130.724698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075131.428785:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075131.428799:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1769075131.428806:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075131.428808:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075131.428819:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1820 00000100:00080000:3.0:1769075131.428830:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075131.428854:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075131.428958:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923573376/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075131.428986:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075131.428989:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075131.428991:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075131.429012:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1820 00000100:00080000:3.0:1769075131.429018:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075131.429025:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075131.429039:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075131.429057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923573376:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075131.429066:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923573504/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075131.429085:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075131.429112:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075131.429116:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075131.429121:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1820 00000100:00080000:3.0:1769075131.429131:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075131.429144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075131.429161:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075131.429173:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075131.429207:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075131/real 1769075131] req@ffff969e0a7fc000 x1855007923573376/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075131.429216:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923573632/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075131.429232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923573376/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075131.429239:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075131.429248:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075131.429249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075131.429254:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075131.429258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075131.429263:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1820 00000100:00100000:2.0:1769075131.429265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923573376:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075131.429272:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075131.429297:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075131.429306:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075131.429319:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075131.429329:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923573760/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075131.429359:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075131.429365:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075131.429370:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075131.429379:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075131.429391:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1820 00000100:00080000:3.0:1769075131.429401:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075131.429402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075131/real 1769075131] req@ffff969e0a7ffb80 x1855007923573504/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075131.429431:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075131.429439:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923573504/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075131.429457:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075131.429458:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075131.429461:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075131.429468:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075131.429475:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573632:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075131.429486:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923573888/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075131.429500:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075131.429524:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075131/real 1769075131] req@ffff969e0a7fd500 x1855007923573632/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075131.429537:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923573632/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075131.429545:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075131.429548:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075131.429551:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573632:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075131.429568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075131.429585:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075131.429627:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075131/real 1769075131] req@ffff969e0a7ff800 x1855007923573760/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075131.429659:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923573760/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075131.429680:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075131.429685:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075131.429689:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075131.429696:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573888:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075131.429724:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075131.429736:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075131/real 1769075131] req@ffff969e0a7fdc00 x1855007923573888/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075131.429750:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923573888/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075131.429760:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075131.429764:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075131.429768:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923573888:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075131.748700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075132.452725:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075132.772649:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075133.476754:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075133.796677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075134.500880:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075134.820766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075135.524747:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075135.844659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075136.548769:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075136.548796:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075136.548803:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075136.548855:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1825 00000100:00080000:0.0:1769075136.548871:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769075136.548875:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075136.548881:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075136.548959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923574016/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075136.549006:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075136.549012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075136.549015:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075136.549021:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1825 00000100:00080000:0.0:1769075136.549026:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075136.549031:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075136.549044:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075136.549073:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923574144/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075136.549099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923574016:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075136.549131:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075136.549147:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075136.549168:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075136.549175:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1825 00000100:00080000:0.0:1769075136.549183:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075136.549190:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075136.549199:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075136.549221:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923574272/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075136.549237:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075136.549243:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075136.549246:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075136.549250:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1825 00000100:00080000:0.0:1769075136.549255:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075136.549262:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075136.549269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075136.549282:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075136.549306:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923574400/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075136.549321:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075136.549331:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769075136.549331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075136/real 1769075136] req@ffff969e06a60000 x1855007923574016/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075136.549337:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075136.549353:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1825 00000100:00080000:0.0:1769075136.549371:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075136.549378:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075136.549384:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075136.549398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923574016/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075136.549405:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923574528/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075136.549424:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075136.549434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075136.549441:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923574016:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075136.549473:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075136.549549:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075136.549578:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075136/real 1769075136] req@ffff969e06a60700 x1855007923574144/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075136.549659:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923574144/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075136.549672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075136.549678:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075136.549685:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075136.549698:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574272:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075136.549744:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075136.549768:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075136/real 1769075136] req@ffff969e06a63800 x1855007923574272/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075136.549853:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923574272/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075136.549865:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075136.549870:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075136.549874:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574272:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075136.549884:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075136.549917:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075136.549932:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075136/real 1769075136] req@ffff969e06a60a80 x1855007923574400/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075136.549947:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923574400/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075136.549958:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075136.549962:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075136.549966:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075136.549973:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574528:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075136.549992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075136.550003:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075136/real 1769075136] req@ffff969e06a61f80 x1855007923574528/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075136.550017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923574528/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075136.550046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075136.550051:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075136.550055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574528:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075136.868761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075137.572766:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075137.892683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075138.596769:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075138.916788:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075139.620822:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075140.644881:0:32714:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1769075140.645335:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075140.645340:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075140.645343:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075140.645366:0:32714:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff969e12057000 lock: ffff969e097e1400/0x44e2946041b60318 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b91fc930f8887d1 expref: -99 pid: 22546 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1769075140.645460:0:32714:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff969e12057000 lock: ffff969e097e1400/0x44e2946041b60318 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b91fc930f8887d1 expref: -99 pid: 22546 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1769075140.645467:0:32714:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1769075140.645502:0:32714:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff969e12057000 lock: ffff969e097e1400/0x44e2946041b60318 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b91fc930f8887d1 expref: -99 pid: 22546 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1769075140.645523:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075140.645525:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075140.645527:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075140.645543:0:2417:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:0.0:1769075140.645567:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff969e06a62a00 x1855007923574656/t0(0) o103->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075141.668770:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075141.668794:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075141.668799:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075141.668811:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1830 00000100:00080000:3.0:1769075141.668826:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075141.668835:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075141.668876:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075141.668898:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923574784/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075141.668934:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075141.668937:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075141.668939:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075141.668947:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1830 00000100:00080000:3.0:1769075141.668950:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075141.668956:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075141.668966:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075141.668980:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923574784:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075141.668980:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923574912/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075141.669005:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075141.669009:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075141.669010:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075141.669012:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1830 00000100:00080000:3.0:1769075141.669054:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075141.669059:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075141.669064:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075141.669077:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923575040/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075141.669085:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075141.669088:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075141.669090:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075141.669106:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1830 00000100:00080000:3.0:1769075141.669118:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075141.669129:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075141.669130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075141.669142:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075141.669165:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923575168/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075141.669177:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075141.669180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075141/real 1769075141] req@ffff969e0a7fc380 x1855007923574784/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075141.669188:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075141.669212:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075141.669219:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1830 00000100:00080000:3.0:1769075141.669225:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075141.669230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923574784/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075141.669237:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075141.669253:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075141.669276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075141.669289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075141.669290:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923575296/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075141.669302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923574784:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075141.669342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574912:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075141.669401:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075141.669412:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075141/real 1769075141] req@ffff969e0a7fca80 x1855007923574912/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075141.669423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923574912/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075141.669430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075141.669433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075141.669435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923574912:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075141.669440:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575040:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075141.669454:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075141.669461:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075141/real 1769075141] req@ffff969e0a7fce00 x1855007923575040/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075141.669470:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923575040/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075141.669476:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075141.669478:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075141.669481:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575040:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075141.669486:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575168:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075141.669498:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075141.669505:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075141/real 1769075141] req@ffff969e0a7fdf80 x1855007923575168/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075141.669527:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923575168/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075141.669533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075141.669536:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075141.669538:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575168:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075141.669543:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575296:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075141.669557:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075141.669565:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075141/real 1769075141] req@ffff969e0a7fe300 x1855007923575296/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075141.669574:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923575296/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075141.669580:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075141.669582:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075141.669585:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575296:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075142.692804:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075143.716840:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075144.740876:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075145.764829:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075146.788912:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075146.788922:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075146.788938:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075146.788944:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075146.788957:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1835 00000100:00080000:0.0:1769075146.788972:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075146.789020:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075146.789059:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923575424/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075146.789094:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075146.789098:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075146.789100:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075146.789116:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1835 00000100:00080000:0.0:1769075146.789121:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075146.789125:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075146.789135:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075146.789147:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923575552/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075146.789179:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075146.789187:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769075146.789189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923575424:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075146.789191:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075146.789208:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1835 00000100:00080000:0.0:1769075146.789221:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075146.789236:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075146.789243:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075146.789278:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923575680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075146.789288:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075146.789292:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075146.789302:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075146.789307:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1835 00000100:00080000:0.0:1769075146.789315:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075146.789319:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075146.789326:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075146.789361:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075146.789369:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923575808/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075146.789382:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075146.789397:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075146.789400:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075146.789424:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1835 00000100:00080000:0.0:1769075146.789429:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075146.789434:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075146.789442:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075146.789487:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075146/real 1769075146] req@ffff969e06a60380 x1855007923575424/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075146.789494:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923575936/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075146.789586:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923575424/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075146.789674:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075146.789709:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075146.789714:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923575424:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075146.789764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575552:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075146.789794:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075146.789809:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075146/real 1769075146] req@ffff969e06a62d80 x1855007923575552/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075146.789822:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923575552/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075146.789831:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075146.789835:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075146.789841:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575552:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075146.789868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575680:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075146.789901:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075146.789920:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075146/real 1769075146] req@ffff969e06a61500 x1855007923575680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075146.789952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923575680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075146.789987:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075146.789994:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075146.790001:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575680:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075146.790029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575808:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075146.790082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075146.790099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075146/real 1769075146] req@ffff969e06a61180 x1855007923575808/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075146.790134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923575808/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075146.790147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075146.790153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075146.790159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575808:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075146.790170:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575936:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075146.790203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075146.790237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075146/real 1769075146] req@ffff969e06a62680 x1855007923575936/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075146.790256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923575936/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075146.790270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075146.790275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075146.790290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923575936:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075147.812853:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075148.836911:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075149.860813:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075150.884761:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075151.908790:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075151.908808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075151.908816:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075151.908842:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1841 00000100:00080000:3.0:1769075151.908878:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075151.908890:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075151.908932:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075151.908966:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923576064/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075151.909019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075151.909025:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075151.909027:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075151.909034:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1841 00000100:00080000:3.0:1769075151.909039:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075151.909045:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075151.909058:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075151.909065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923576064:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075151.909081:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923576192/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075151.909119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075151.909124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075151.909127:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075151.909133:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1841 00000100:00080000:3.0:1769075151.909157:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075151.909160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075151.909167:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075151.909209:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075151/real 1769075151] req@ffff969e0a7fed80 x1855007923576064/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075151.909220:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075151.909233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923576064/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075151.909241:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923576320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075151.909257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075151.909270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075151.909270:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075151.909283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1769075151.909285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923576064:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075151.909287:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075151.909306:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1841 00000100:00080000:3.0:1769075151.909312:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075151.909316:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075151.909323:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075151.909329:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576192:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075151.909336:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923576448/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075151.909347:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075151.909357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075151.909362:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075151.909366:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075151.909372:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1841 00000100:00080000:3.0:1769075151.909376:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075151.909378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075151/real 1769075151] req@ffff969e0a7fe680 x1855007923576192/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075151.909387:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075151.909403:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075151.909410:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923576192/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075151.909430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075151.909440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075151.909441:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923576576/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075151.909469:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576192:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075151.909509:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576320:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075151.909536:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075151.909546:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075151/real 1769075151] req@ffff969e0a7fc700 x1855007923576320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075151.909668:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923576320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075151.909682:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075151.909687:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075151.909691:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576320:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075151.909709:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576448:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075151.909745:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075151.909756:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075151/real 1769075151] req@ffff969e0a7fd180 x1855007923576448/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075151.909767:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923576448/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075151.909775:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075151.909779:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075151.909782:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576448:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075151.909787:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576576:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075151.909801:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075151.909821:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075151/real 1769075151] req@ffff969e0a7fea00 x1855007923576576/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075151.909848:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923576576/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075151.909865:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075151.909867:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075151.909869:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576576:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075152.932842:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075153.956793:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075154.980798:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075156.004776:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075157.028686:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075157.028701:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075157.028735:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075157.028761:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1846 00000100:00080000:3.0:1769075157.028776:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075157.028787:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075157.028804:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075157.028855:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923576704/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075157.028909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075157.028914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075157.028917:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075157.028923:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1846 00000100:00080000:3.0:1769075157.028930:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075157.028937:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075157.028952:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075157.028964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923576704:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075157.028993:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923576832/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075157.029006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075157.029010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075157.029013:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075157.029017:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1846 00000100:00080000:3.0:1769075157.029031:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075157.029035:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075157.029040:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075157.029056:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923576960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075157.029114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075157.029122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075157.029126:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075157.029136:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1846 00000100:00080000:3.0:1769075157.029145:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075157.029159:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075157.029160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075157.029186:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075157.029212:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923577088/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075157.029226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075157.029231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075157.029235:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075157.029250:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1846 00000100:00080000:3.0:1769075157.029255:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075157.029260:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075157.029267:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075157.029281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075157/real 1769075157] req@ffff969e0a7fd880 x1855007923576704/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075157.029285:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923577216/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075157.029374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923576704/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075157.029392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075157.029399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075157.029405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923576704:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075157.029456:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576832:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075157.029517:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075157.029544:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075157/real 1769075157] req@ffff969e0a7fea00 x1855007923576832/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075157.029557:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923576832/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075157.029566:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075157.029569:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075157.029573:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576832:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075157.029579:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576960:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075157.029641:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075157.029653:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075157/real 1769075157] req@ffff969e0a7fd180 x1855007923576960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075157.029665:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923576960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075157.029673:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075157.029677:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075157.029680:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923576960:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075157.029686:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577088:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075157.029703:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075157.029711:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075157/real 1769075157] req@ffff969e0a7fc700 x1855007923577088/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075157.029722:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923577088/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075157.029738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075157.029741:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075157.029774:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577088:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075157.029782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577216:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075157.029806:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075157.029818:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075157/real 1769075157] req@ffff969e0a7fe680 x1855007923577216/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075157.029832:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923577216/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075157.029842:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075157.029845:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075157.029849:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577216:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075158.052869:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075159.076826:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075160.100834:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075161.124869:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075162.148679:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075162.148696:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075162.148703:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075162.148719:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1851 00010000:00010000:2.0:1769075162.148766:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075162.148768:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075162.148786:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075162.148838:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923577344/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075162.148871:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075162.148875:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075162.148877:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075162.148881:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1851 00000100:00080000:3.0:1769075162.148885:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075162.148888:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075162.148900:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075162.148911:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923577472/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075162.148932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075162.148933:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923577344:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075162.148947:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075162.148951:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075162.148956:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1851 00000100:00080000:3.0:1769075162.148959:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075162.148962:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075162.148968:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075162.148981:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923577600/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075162.149007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075162.149019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075162.149023:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075162.149027:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1851 00000100:00080000:3.0:1769075162.149032:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075162.149038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075162.149045:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075162.149079:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075162.149093:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923577728/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075162.149105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075162.149109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075162.149113:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075162.149116:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1851 00000100:00080000:3.0:1769075162.149123:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075162.149128:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075162.149137:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075162.149149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075162/real 1769075162] req@ffff969e0a7fed80 x1855007923577344/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075162.149155:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923577856/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075162.149238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923577344/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075162.149265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075162.149270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075162.149274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923577344:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075162.149323:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577472:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075162.149356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075162.149371:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075162/real 1769075162] req@ffff969e0a7fe300 x1855007923577472/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075162.149445:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923577472/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075162.149453:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075162.149457:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075162.149460:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577472:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075162.149466:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577600:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075162.149485:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075162.149509:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075162/real 1769075162] req@ffff969e0a7fdf80 x1855007923577600/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075162.149525:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923577600/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075162.149537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075162.149542:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075162.149546:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577600:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075162.149566:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577728:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075162.149634:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075162.149647:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075162/real 1769075162] req@ffff969e0a7fce00 x1855007923577728/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075162.149660:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923577728/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075162.149671:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075162.149676:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075162.149681:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577728:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075162.149689:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577856:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075162.149717:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075162.149733:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075162/real 1769075162] req@ffff969e0a7fca80 x1855007923577856/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075162.149749:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923577856/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075162.149760:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075162.149764:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075162.149767:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923577856:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075163.172756:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075164.196840:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075165.220833:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075166.244839:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075167.268763:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075167.268781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075167.268789:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075167.268805:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1856 00000100:00080000:3.0:1769075167.268825:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075167.268836:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075167.268897:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075167.268918:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923577984/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075167.268979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075167.268983:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075167.268986:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075167.268991:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1856 00000100:00080000:3.0:1769075167.268996:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075167.269002:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075167.269015:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075167.269055:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923578112/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075167.269056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923577984:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075167.269100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075167.269105:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075167.269127:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075167.269136:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1856 00000100:00080000:3.0:1769075167.269142:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075167.269149:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075167.269158:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075167.269187:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923578240/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075167.269203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075167.269209:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075167.269212:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075167.269217:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1856 00000100:00080000:3.0:1769075167.269234:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075167.269243:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075167.269252:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075167.269268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578112:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075167.269279:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923578368/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075167.269329:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075167.269353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075167.269356:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075167.269364:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1856 00000100:00100000:2.0:1769075167.269369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578240:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075167.269375:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075167.269390:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075167.269400:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075167.269421:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923578496/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075167.269429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578368:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075167.269510:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578496:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075168.292982:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075169.316830:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075170.341139:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769075170.341405:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769075170.341475:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769075170.341506:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075170.341581:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075170.341710:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075170.341712:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075167/real 1769075170] req@ffff969e0a7fc380 x1855007923577984/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1769075170.341729:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075170.341745:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075170.341800:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923577984/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075170.341824:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075170.341834:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075170.341843:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923577984:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075170.341880:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075167/real 1769075170] req@ffff969e0a7ffb80 x1855007923578496/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075170.341907:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923578496/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075170.341925:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075170.341933:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075170.341973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578496:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075170.341988:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075167/real 1769075170] req@ffff969e0a7fd500 x1855007923578368/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075170.342014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923578368/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075170.342047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075170.342055:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075170.342063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578368:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075170.342077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075167/real 1769075170] req@ffff969e0a7fdc00 x1855007923578112/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075170.342104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923578112/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075170.342120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075170.342127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075170.342134:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578112:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075170.342147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075167/real 1769075170] req@ffff969e0a7ff800 x1855007923578240/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075170.342172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923578240/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075170.342188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075170.342196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075170.342219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578240:192.168.204.143@tcp:38:kworker.0 00010000:00010000:2.0:1769075171.365008:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075172.388740:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075172.388754:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075172.388758:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075172.388771:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1861 00000100:00080000:3.0:1769075172.388785:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075172.388794:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075172.388840:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923578624/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075172.388873:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075172.388904:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075172.388906:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075172.388910:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1861 00000100:00080000:3.0:1769075172.388914:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075172.388918:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075172.388930:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075172.388933:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923578624:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075172.388949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923578752/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075172.388969:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075172.388974:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075172.388978:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769075172.388980:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075172.388988:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1861 00000100:00080000:3.0:1769075172.388998:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075172.389006:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075172.389015:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075172.389030:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923578880/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075172.389041:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075172.389046:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075172.389048:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075172.389052:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1861 00000100:00080000:3.0:1769075172.389057:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769075172.389064:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075172.389066:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075172.389082:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075172.389099:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923579008/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075172.389111:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075172.389114:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075172.389117:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075172.389123:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075172/real 1769075172] req@ffff969e0a7fc000 x1855007923578624/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075172.389125:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1861 00000100:00080000:3.0:1769075172.389132:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075172.389142:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075172.389149:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075172.389159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923578624/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075172.389164:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923579136/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075172.389188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075172.389196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075172.389202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923578624:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075172.389254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578752:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075172.389281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075172.389298:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075172/real 1769075172] req@ffff969e0a7ff800 x1855007923578752/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075172.389312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923578752/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075172.389321:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075172.389324:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075172.389327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578752:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075172.389335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578880:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075172.389356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075172.389367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075172/real 1769075172] req@ffff969e0a7fdc00 x1855007923578880/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075172.389398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923578880/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075172.389406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075172.389410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075172.389414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923578880:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075172.389421:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579008:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075172.389444:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075172.389455:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075172/real 1769075172] req@ffff969e0a7fd500 x1855007923579008/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075172.389468:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923579008/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075172.389476:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075172.389480:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075172.389483:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579008:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075172.389489:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579136:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075172.389506:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075172.389515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075172/real 1769075172] req@ffff969e0a7ffb80 x1855007923579136/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075172.389526:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923579136/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075172.389547:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075172.389551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075172.389555:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579136:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075173.412954:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075174.436713:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075175.460807:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075176.484829:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075177.508691:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075177.508700:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075177.508703:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075177.508713:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1866 00000100:00080000:0.0:1769075177.508722:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075177.508728:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075177.508757:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075177.508760:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923579264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075177.508784:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075177.508787:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075177.508788:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075177.508791:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1866 00000100:00080000:0.0:1769075177.508794:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075177.508796:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075177.508803:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075177.508812:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923579392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075177.508821:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075177.508825:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769075177.508825:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923579264:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075177.508843:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075177.508848:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1866 00000100:00080000:0.0:1769075177.508853:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075177.508867:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075177.508873:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075177.508889:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923579520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075177.508899:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075177.508904:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1769075177.508904:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075177.508909:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075177.508913:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1866 00000100:00080000:0.0:1769075177.508917:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075177.508921:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075177.508936:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075177.508946:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923579648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075177.508953:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075177.508954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075177/real 1769075177] req@ffff969e06a63100 x1855007923579264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075177.508960:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075177.508963:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075177.508971:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1866 00000100:00080000:0.0:1769075177.508976:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075177.508979:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075177.508982:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075177.508985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923579264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075177.508999:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769075177.509002:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923579776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075177.509006:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075177.509013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923579264:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075177.509054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579392:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075177.509073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075177.509082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075177/real 1769075177] req@ffff969e06a61880 x1855007923579392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075177.509090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923579392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075177.509096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075177.509105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075177.509108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579392:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075177.509112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579520:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075177.509150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075177.509160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075177/real 1769075177] req@ffff969e06a62680 x1855007923579520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075177.509168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923579520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075177.509174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075177.509176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075177.509179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579520:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075177.509183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579648:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075177.509195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075177.509201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075177/real 1769075177] req@ffff969e06a61180 x1855007923579648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075177.509208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923579648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075177.509214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075177.509216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075177.509218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579648:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075177.509221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579776:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075177.509240:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075177.509247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075177/real 1769075177] req@ffff969e06a61500 x1855007923579776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075177.509254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923579776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075177.509260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075177.509262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075177.509263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923579776:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075178.532822:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075179.556842:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075180.580816:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075181.604808:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075182.628738:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075182.628756:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075182.628762:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075182.628775:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1871 00000100:00080000:0.0:1769075182.628789:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075182.628799:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075182.628829:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075182.628847:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923579904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075182.628914:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075182.628919:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075182.628921:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075182.628926:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1871 00000100:00080000:0.0:1769075182.628931:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075182.628937:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075182.628949:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075182.628986:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923579904:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075182.628989:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923580032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075182.629015:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075182.629019:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075182.629022:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075182.629027:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1871 00000100:00080000:0.0:1769075182.629031:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075182.629082:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075182.629090:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075182.629107:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923580160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075182.629123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075182.629128:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075182.629135:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075182.629139:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075182.629147:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1871 00000100:00080000:0.0:1769075182.629157:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075182.629165:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075182.629177:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075182.629178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075182/real 1769075182] req@ffff969e06a62d80 x1855007923579904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075182.629227:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923580288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075182.629230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923579904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075182.629283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075182.629288:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075182.629295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075182.629299:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075182.629306:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075182.629308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923579904:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075182.629319:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1871 00000100:00080000:0.0:1769075182.629329:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075182.629335:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075182.629344:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075182.629355:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580032:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075182.629366:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923580416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075182.629417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075182.629438:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075182/real 1769075182] req@ffff969e06a60380 x1855007923580032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075182.629453:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923580032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075182.629462:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075182.629466:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075182.629470:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580032:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075182.629477:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580160:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075182.629496:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075182.629523:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075182/real 1769075182] req@ffff969e06a61f80 x1855007923580160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075182.629538:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923580160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075182.629548:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075182.629552:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075182.629556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580160:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075182.629563:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580288:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075182.629584:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075182.629595:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075182/real 1769075182] req@ffff969e06a60a80 x1855007923580288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075182.629676:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923580288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075182.629687:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075182.629692:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075182.629696:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580288:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075182.629718:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580416:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075182.629758:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075182.629779:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075182/real 1769075182] req@ffff969e06a63800 x1855007923580416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075182.629804:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923580416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075182.629823:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075182.629830:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075182.629837:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580416:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075183.652787:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075184.676753:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075185.700717:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075187.748757:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075187.748778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075187.748788:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075187.748809:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1876 00010000:00010000:2.0:1769075187.748839:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075187.748845:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075187.748868:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075187.749017:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923580544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075187.749052:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075187.749058:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075187.749060:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075187.749068:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1876 00000100:00080000:3.0:1769075187.749107:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075187.749110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923580544:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075187.749119:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075187.749136:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075187.749160:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923580672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075187.749177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075187.749182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1769075187.749188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075187.749191:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075187.749202:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1876 00000100:00080000:3.0:1769075187.749213:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075187.749220:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075187.749241:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075187.749251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075187/real 1769075187] req@ffff969e0a7fc380 x1855007923580544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075187.749262:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923580800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075187.749278:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075187.749306:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075187.749317:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075187.749330:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1876 00000100:00080000:3.0:1769075187.749336:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075187.749338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923580544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075187.749351:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769075187.749358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075187.749363:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075187.749366:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075187.749376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923580544:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075187.749390:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923580928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075187.749402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580672:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075187.749425:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075187.749426:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075187.749434:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075187.749438:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075187.749439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075187/real 1769075187] req@ffff969e0a7fca80 x1855007923580672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075187.749458:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1876 00000100:00100000:2.0:1769075187.749463:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923580672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075187.749469:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075187.749471:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075187.749479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075187.749484:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075187.749488:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580672:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075187.749499:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075187.749500:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580800:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075187.749529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075187.749535:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923581056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075187.749540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075187/real 1769075187] req@ffff969e0a7fce00 x1855007923580800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075187.749569:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923580800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075187.749586:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075187.749626:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075187.749630:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580800:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075187.749644:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580928:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075187.749662:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075187.749670:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075187/real 1769075187] req@ffff969e0a7fdf80 x1855007923580928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075187.749679:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923580928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075187.749685:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075187.749687:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075187.749689:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923580928:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075187.749693:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581056:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075187.749703:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075187.749709:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075187/real 1769075187] req@ffff969e0a7fe300 x1855007923581056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075187.749715:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923581056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075187.749721:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075187.749723:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075187.749732:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581056:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075189.796842:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075191.844814:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075192.868740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075192.868758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075192.868765:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075192.868782:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1882 00000100:00080000:3.0:1769075192.868817:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075192.868828:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075192.868944:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923581184/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075192.868975:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075192.868978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075192.868980:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075192.868984:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1882 00000100:00080000:3.0:1769075192.868988:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075192.868991:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075192.869027:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075192.869039:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923581184:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075192.869046:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923581312/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075192.869061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075192.869065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075192.869068:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075192.869073:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1882 00000100:00080000:3.0:1769075192.869077:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075192.869082:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075192.869088:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075192.869101:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923581440/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075192.869108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075192.869111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075192.869112:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075192.869115:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1882 00000100:00080000:3.0:1769075192.869128:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075192.869130:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075192.869134:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075192.869140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075192.869145:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923581568/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075192.869152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075192.869155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075192.869162:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075192.869166:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1882 00000100:00080000:3.0:1769075192.869172:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075192.869181:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075192.869186:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075192.869195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075192/real 1769075192] req@ffff969e0a7fed80 x1855007923581184/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075192.869198:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923581696/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075192.869239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923581184/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075192.869254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075192.869260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075192.869265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923581184:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075192.869299:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581312:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075192.869326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075192.869351:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075192/real 1769075192] req@ffff969e0a7fe680 x1855007923581312/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075192.869368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923581312/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075192.869396:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075192.869401:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075192.869405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581312:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075192.869414:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581440:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075192.869468:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075192.869481:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075192/real 1769075192] req@ffff969e0a7fc700 x1855007923581440/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075192.869495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923581440/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075192.869505:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075192.869509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075192.869513:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581440:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075192.869520:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581568:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075192.869545:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075192.869557:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075192/real 1769075192] req@ffff969e0a7fd180 x1855007923581568/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075192.869571:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923581568/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075192.869580:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075192.869584:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075192.869588:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581568:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075192.869594:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581696:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075192.869615:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075192.869678:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075192/real 1769075192] req@ffff969e0a7fea00 x1855007923581696/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075192.869710:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923581696/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075192.869720:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075192.869724:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075192.869728:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581696:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075193.892796:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075195.940811:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075197.988742:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075197.988759:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075197.988767:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075197.988816:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1887 00000100:00080000:3.0:1769075197.988837:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075197.988849:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075197.988866:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075197.988917:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923581824/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075197.988973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075197.988981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075197.988985:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075197.988997:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1887 00000100:00080000:3.0:1769075197.989007:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075197.989016:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075197.989029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923581824:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075197.989037:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075197.989075:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923581952/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075197.989130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075197.989134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075197.989137:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075197.989149:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1887 00000800:00000100:2.0:1769075197.989150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075197.989162:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075197.989172:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075197.989182:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075197.989202:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923582080/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075197.989217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075197/real 1769075197] req@ffff969e0a7fd880 x1855007923581824/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075197.989218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075197.989234:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075197.989239:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075197.989251:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1887 00000100:00080000:3.0:1769075197.989258:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075197.989264:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075197.989271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923581824/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075197.989275:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075197.989302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769075197.989303:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923582208/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075197.989318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075197.989334:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075197.989337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923581824:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075197.989346:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075197.989353:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075197.989360:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1887 00000100:00080000:3.0:1769075197.989373:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075197.989380:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075197.989388:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581952:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075197.989393:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075197.989416:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923582336/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075197.989456:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075197.989488:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075197/real 1769075197] req@ffff969e0a7fea00 x1855007923581952/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075197.989550:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923581952/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075197.989568:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075197.989576:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075197.989581:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923581952:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075197.989633:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582080:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075197.989668:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075197.989684:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075197/real 1769075197] req@ffff969e0a7fd180 x1855007923582080/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075197.989701:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923582080/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075197.989712:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075197.989716:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075197.989720:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582080:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075197.989728:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582208:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075197.989749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075197.989760:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075197/real 1769075197] req@ffff969e0a7fc700 x1855007923582208/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075197.989775:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923582208/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075197.989837:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075197.989843:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075197.989847:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582208:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075197.989872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582336:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075197.989903:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075197.989917:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075197/real 1769075197] req@ffff969e0a7fe680 x1855007923582336/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075197.989931:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923582336/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075197.989943:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075197.989947:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075197.989951:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582336:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075200.036793:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075202.084956:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075203.108758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075203.108777:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075203.108785:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075203.108802:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1892 00000100:00080000:3.0:1769075203.108820:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075203.108832:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075203.108903:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923582464/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075203.108943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075203.108948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075203.108951:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075203.108957:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1892 00000100:00080000:3.0:1769075203.108962:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075203.108968:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075203.108999:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075203.109016:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923582592/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075203.109035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075203.109038:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923582464:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075203.109066:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075203.109070:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075203.109078:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1892 00000100:00080000:3.0:1769075203.109083:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075203.109088:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075203.109095:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075203.109116:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923582720/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075203.109130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075203.109135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075203.109137:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075203.109145:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1892 00000100:00080000:3.0:1769075203.109168:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075203.109173:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075203.109180:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075203.109190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075203.109205:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923582848/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075203.109224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075203.109228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075203.109231:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075203.109238:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1892 00000100:00080000:3.0:1769075203.109261:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075203.109270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075203/real 1769075203] req@ffff969e0a7fed80 x1855007923582464/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075203.109271:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075203.109303:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075203.109313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923582464/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075203.109330:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923582976/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075203.109335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075203.109366:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075203.109379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923582464:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075203.109417:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582592:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075203.109454:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075203.109491:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075203/real 1769075203] req@ffff969e0a7fe300 x1855007923582592/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075203.109540:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923582592/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075203.109560:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075203.109567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075203.109575:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582592:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075203.109680:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582720:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075203.109758:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075203.109776:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075203/real 1769075203] req@ffff969e0a7fdf80 x1855007923582720/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075203.109794:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923582720/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075203.109806:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075203.109810:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075203.109815:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582720:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075203.109825:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582848:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075203.109881:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075203.109912:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075203/real 1769075203] req@ffff969e0a7fce00 x1855007923582848/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075203.109939:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923582848/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075203.109957:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075203.109975:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075203.109982:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582848:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075203.109995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582976:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075203.110037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075203.110061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075203/real 1769075203] req@ffff969e0a7fca80 x1855007923582976/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075203.110088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923582976/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075203.110105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075203.110112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075203.110119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923582976:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075204.132809:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075206.180802:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075208.228754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075208.228771:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075208.228777:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769075208.228823:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075208.228981:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1897 00000100:00080000:2.0:1769075208.229008:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075208.229015:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075208.229047:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c700 x1855007923583104/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075208.229072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075208.229075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075208.229077:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075208.229081:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1897 00000100:00080000:2.0:1769075208.229085:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075208.229088:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075208.229096:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075208.229104:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c000 x1855007923583232/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075208.229115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769075208.229116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923583104:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075208.229120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075208.229127:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075208.229133:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1897 00000100:00080000:2.0:1769075208.229135:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075208.229138:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075208.229141:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075208.229150:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923583360/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075208.229176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075208.229178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075208.229180:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075208.229184:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1897 00000100:00080000:2.0:1769075208.229188:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075208.229190:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075208.229194:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075208.229203:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923583488/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075208.229209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075208.229211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075208.229213:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769075208.229222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075208.229228:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1897 00000100:00080000:2.0:1769075208.229233:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075208.229237:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075208.229241:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075208.229252:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923583616/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075208.229283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075208/real 1769075208] req@ffff969e0a46c700 x1855007923583104/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075208.229301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c700 x1855007923583104/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075208.229310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075208.229313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075208.229317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923583104:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075208.229342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583232:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075208.229359:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075208.229367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075208/real 1769075208] req@ffff969e0a46c000 x1855007923583232/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075208.229376:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c000 x1855007923583232/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075208.229382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075208.229384:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075208.229386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583232:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075208.229399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583360:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075208.229411:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075208.229417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075208/real 1769075208] req@ffff969e0a46f100 x1855007923583360/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075208.229437:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923583360/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075208.229443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075208.229446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075208.229448:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583360:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075208.229452:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583488:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075208.229467:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075208.229474:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075208/real 1769075208] req@ffff969e0a46c380 x1855007923583488/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075208.229482:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923583488/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075208.229487:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075208.229489:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075208.229491:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583488:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075208.229494:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583616:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075208.229505:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075208.229511:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075208/real 1769075208] req@ffff969e0a46f800 x1855007923583616/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075208.229518:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923583616/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075208.229533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075208.229535:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075208.229537:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583616:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075210.276829:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075212.324935:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075213.349029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923583744:192.168.204.143@tcp:250:kworker.0 00000800:00000100:0.0:1769075213.349118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075213.349163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075213/real 1769075213] req@ffff969e10b63800 x1855007923583744/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075213.349192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b63800 x1855007923583744/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075213.349220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075213.349225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075213.349230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923583744:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075213.349236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583872:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075213.349254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075213.349264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075213/real 1769075213] req@ffff969e10b60700 x1855007923583872/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075213.349275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923583872/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075213.349283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075213.349293:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075213.349297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923583872:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075213.349326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584000:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075213.349342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075213.349349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075213/real 1769075213] req@ffff969e10b62d80 x1855007923584000/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075213.349357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923584000/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075213.349363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075213.349365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075213.349367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584000:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075213.349371:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584128:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075213.349382:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075213.349387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075213/real 1769075213] req@ffff969e10b60000 x1855007923584128/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075213.349394:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923584128/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075213.349401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075213.349403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075213.349405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584128:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075213.349408:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584256:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075213.349427:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075213.349434:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075213/real 1769075213] req@ffff969e10b61180 x1855007923584256/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075213.349442:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923584256/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075213.349448:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075213.349449:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075213.349451:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584256:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075214.372850:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075216.420974:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075218.468977:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923584384:192.168.204.143@tcp:250:kworker.0 00000800:00000100:0.0:1769075218.469105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075218.469171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075218/real 1769075218] req@ffff969e10b60380 x1855007923584384/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075218.469202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923584384/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075218.469225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075218.469236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075218.469243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923584384:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075218.469277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584512:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075218.469305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075218.469319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075218/real 1769075218] req@ffff969e10b61180 x1855007923584512/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075218.469346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923584512/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075218.469354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075218.469357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075218.469360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584512:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075218.469367:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584640:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075218.469383:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075218.469392:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075218/real 1769075218] req@ffff969e10b60000 x1855007923584640/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075218.469402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923584640/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1769075218.469407:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075218.469408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075218.469413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075218.469415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584640:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075218.469420:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584768:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075218.469433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075218.469470:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075218/real 1769075218] req@ffff969e10b62d80 x1855007923584768/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075218.469478:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923584768/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075218.469495:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075218.469498:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075218.469500:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584768:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075218.469505:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584896:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075218.469522:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075218.469529:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075218/real 1769075218] req@ffff969e10b60700 x1855007923584896/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075218.469536:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923584896/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075218.469542:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075218.469544:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075218.469546:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923584896:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075220.516748:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075222.564841:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075223.588686:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075223.588698:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075223.588703:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075223.588715:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1912 00000100:00080000:3.0:1769075223.588726:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075223.588734:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075223.588767:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923585024/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075223.588792:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075223.588797:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075223.588799:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075223.588803:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1912 00000100:00080000:3.0:1769075223.588807:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075223.588811:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075223.588822:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075223.588839:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923585152/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075223.588849:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075223.588854:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075223.588856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075223.588860:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1912 00000100:00100000:0.0:1769075223.588876:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923585024:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075223.588886:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075223.588894:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075223.588901:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075223.588914:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923585280/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075223.588923:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075223.588926:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075223.588929:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075223.588932:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1912 00000100:00080000:3.0:1769075223.588938:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075223.588940:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075223.588944:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075223.588951:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923585408/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075223.588957:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075223.588959:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075223.588961:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075223.588963:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1912 00000100:00080000:3.0:1769075223.588967:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769075223.588974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075223.588979:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075223.588985:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075223.588994:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923585536/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075223.589037:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075223/real 1769075223] req@ffff969e0a7fc380 x1855007923585024/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075223.589063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923585024/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075223.589077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075223.589083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075223.589086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923585024:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075223.589108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585152:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075223.589133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075223.589148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075223/real 1769075223] req@ffff969e0a7ffb80 x1855007923585152/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075223.589162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923585152/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075223.589173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075223.589177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075223.589181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585152:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075223.589213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585280:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075223.589236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075223.589249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075223/real 1769075223] req@ffff969e0a7fd500 x1855007923585280/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075223.589263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923585280/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075223.589274:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075223.589276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075223.589279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585280:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075223.589284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585408:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075223.589301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075223.589309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075223/real 1769075223] req@ffff969e0a7fdc00 x1855007923585408/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075223.589336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923585408/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075223.589346:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075223.589365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075223.589370:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585408:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075223.589377:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585536:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075223.589403:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075223.589414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075223/real 1769075223] req@ffff969e0a7ff800 x1855007923585536/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075223.589423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923585536/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075223.589430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075223.589433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075223.589435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585536:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075224.612781:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075226.660759:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075228.708664:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075228.708677:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075228.708694:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075228.708707:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1917 00000100:00080000:3.0:1769075228.708721:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075228.708731:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075228.708736:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075228.708797:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923585664/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075228.708838:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075228.708846:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075228.708848:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075228.708851:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1917 00000100:00080000:3.0:1769075228.708855:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075228.708859:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075228.708869:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075228.708882:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923585792/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075228.708890:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075228.708893:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075228.708894:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075228.708900:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1917 00000100:00100000:0.0:1769075228.708902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923585664:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075228.708903:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075228.708907:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075228.708941:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075228.708952:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923585920/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075228.708959:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075228.708962:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075228.708964:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075228.708966:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1917 00000100:00080000:3.0:1769075228.708969:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075228.708972:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075228.708975:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075228.708982:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923586048/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075228.708988:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075228.708991:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075228.708993:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075228.709007:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1917 00000100:00080000:3.0:1769075228.709009:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075228.709011:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075228.709014:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769075228.709020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075228.709024:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923586176/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075228.709063:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075228/real 1769075228] req@ffff969e0a7fc000 x1855007923585664/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075228.709080:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923585664/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075228.709111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075228.709115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075228.709118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923585664:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075228.709143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585792:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075228.709172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075228.709182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075228/real 1769075228] req@ffff969e0a7ff800 x1855007923585792/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075228.709190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923585792/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075228.709196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075228.709199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075228.709201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585792:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075228.709205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585920:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769075228.709215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075228.709221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075228/real 1769075228] req@ffff969e0a7fdc00 x1855007923585920/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075228.709229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923585920/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075228.709235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075228.709237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075228.709239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923585920:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075228.709249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586048:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075228.709287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075228.709296:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075228/real 1769075228] req@ffff969e0a7fd500 x1855007923586048/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075228.709305:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923586048/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075228.709311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075228.709313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075228.709316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586048:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075228.709320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586176:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769075228.709344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075228.709349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075228/real 1769075228] req@ffff969e0a7ffb80 x1855007923586176/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075228.709357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923586176/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075228.709363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075228.709365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075228.709367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586176:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075230.756768:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075232.804762:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075233.828669:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075233.828681:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075233.828703:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075233.828714:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1923 00000100:00080000:3.0:1769075233.828725:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075233.828731:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075233.828765:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923586304/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075233.828792:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075233.828794:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075233.828796:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075233.828799:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1923 00000100:00080000:3.0:1769075233.828802:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075233.828805:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075233.828813:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075233.828823:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923586432/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075233.828831:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075233.828833:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075233.828834:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769075233.828838:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923586304:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075233.828839:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1923 00000100:00080000:3.0:1769075233.828845:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075233.828849:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075233.828852:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075233.828862:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923586560/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075233.828878:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075233.828881:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075233.828882:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075233.828885:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1923 00000100:00080000:3.0:1769075233.828890:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075233.828892:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075233.828896:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075233.828904:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923586688/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075233.828910:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075233.828913:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075233.828915:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075233.828918:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1923 00000100:00080000:3.0:1769075233.828921:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075233.828924:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075233.828928:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586432:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075233.828928:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075233.828940:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923586816/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769075233.828976:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586560:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075233.828990:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586688:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075233.829006:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586816:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075234.852824:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769075236.901229:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00010000:00010000:0.0:1769075236.901294:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769075236.901314:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769075236.901345:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075236.901418:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075236.901455:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075236.901478:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075236.901506:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:0.0:1769075236.901559:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075233/real 1769075236] req@ffff969e0a7fca80 x1855007923586432/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075236.901687:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923586432/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075236.901727:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075236.901739:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075236.901749:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586432:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075236.901766:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075233/real 1769075236] req@ffff969e0a7fce00 x1855007923586560/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075236.901791:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923586560/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075236.901808:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075236.901816:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075236.901823:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586560:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075236.901839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075233/real 1769075236] req@ffff969e0a7fdf80 x1855007923586688/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075236.901919:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923586688/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075236.901936:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075236.901943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075236.901949:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586688:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769075236.901965:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075233/real 1769075236] req@ffff969e0a7fc380 x1855007923586304/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075236.901990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923586304/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075236.902005:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075236.902011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075236.902018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923586304:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075236.902068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075233/real 1769075236] req@ffff969e0a7fe300 x1855007923586816/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075236.902091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923586816/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075236.902107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075236.902113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075236.902119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923586816:192.168.204.143@tcp:8:kworker.0 00000100:00080000:3.0:1769075238.948704:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075238.948715:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075238.948720:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075238.948731:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1928 00000100:00080000:3.0:1769075238.948759:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075238.948767:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075238.948806:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923586944/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075238.948834:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075238.948837:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075238.948839:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075238.948843:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1928 00000100:00080000:3.0:1769075238.948845:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075238.948848:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075238.948858:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075238.948869:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923587072/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075238.948873:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075238.948882:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075238.948885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923586944:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075238.948890:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075238.948894:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075238.948902:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1928 00000100:00080000:3.0:1769075238.948907:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075238.948912:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075238.948918:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075238.948932:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923587200/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075238.948940:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075238.948949:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075238.948951:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075238.948954:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1928 00000100:00080000:3.0:1769075238.948966:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075238.948969:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075238.948974:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075238.948979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075238.948988:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923587328/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075238.948996:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075238.948998:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075238.948999:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075238.949002:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1928 00000100:00080000:3.0:1769075238.949007:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075238.949009:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075238.949012:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075238.949020:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923587456/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075238.949027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075238/real 1769075238] req@ffff969e0a7fed80 x1855007923586944/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075238.949046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923586944/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075238.949055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075238.949059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075238.949063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923586944:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075238.949088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587072:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075238.949128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075238.949140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075238/real 1769075238] req@ffff969e0a7fe680 x1855007923587072/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075238.949162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923587072/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075238.949168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075238.949170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075238.949173:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587072:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075238.949178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587200:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075238.949194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075238.949205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075238/real 1769075238] req@ffff969e0a7fc700 x1855007923587200/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075238.949218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923587200/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075238.949227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075238.949230:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075238.949234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587200:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075238.949243:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587328:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075238.949270:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075238.949281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075238/real 1769075238] req@ffff969e0a7fd180 x1855007923587328/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075238.949293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923587328/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075238.949315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075238.949319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075238.949323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587328:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075238.949330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587456:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075238.949353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075238.949364:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075238/real 1769075238] req@ffff969e0a7fea00 x1855007923587456/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075238.949376:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923587456/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075238.949386:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075238.949389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075238.949393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587456:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075240.996772:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075243.044736:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075244.068708:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075244.068717:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075244.068721:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075244.068730:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1933 00000100:00080000:3.0:1769075244.068741:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075244.068747:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075244.068781:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923587584/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075244.068821:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075244.068824:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075244.068825:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075244.068828:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1933 00000100:00080000:3.0:1769075244.068831:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075244.068835:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075244.068845:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075244.068859:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923587712/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075244.068871:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075244.068875:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075244.068878:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075244.068883:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1933 00000100:00080000:3.0:1769075244.068887:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075244.068893:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075244.068899:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075244.068907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923587584:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075244.068916:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923587840/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075244.068933:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075244.068938:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075244.068941:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075244.068946:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1933 00000100:00080000:3.0:1769075244.068960:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075244.068966:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075244.068973:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075244.068999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075244.069017:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923587968/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075244.069035:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075244.069039:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075244.069041:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075244.069047:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1933 00000100:00080000:3.0:1769075244.069053:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075244.069062:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075244/real 1769075244] req@ffff969e0a7fd880 x1855007923587584/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075244.069063:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075244.069083:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075244.069092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923587584/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075244.069102:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923588096/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075244.069103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075244.069113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075244.069123:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923587584:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075244.069144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587712:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075244.069165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075244.069174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075244/real 1769075244] req@ffff969e0a7fea00 x1855007923587712/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075244.069183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923587712/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075244.069188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075244.069190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075244.069192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587712:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075244.069215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587840:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075244.069226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075244.069232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075244/real 1769075244] req@ffff969e0a7fd180 x1855007923587840/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075244.069239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923587840/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075244.069244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075244.069246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075244.069248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587840:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075244.069252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587968:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075244.069263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075244.069269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075244/real 1769075244] req@ffff969e0a7fc700 x1855007923587968/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075244.069283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923587968/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075244.069289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075244.069291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075244.069293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923587968:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075244.069297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588096:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075244.069314:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075244.069324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075244/real 1769075244] req@ffff969e0a7fe680 x1855007923588096/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075244.069337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923588096/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075244.069346:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075244.069350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075244.069354:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588096:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075245.092794:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075247.140793:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075249.188654:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075249.188664:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075249.188669:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075249.188680:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1938 00000100:00080000:3.0:1769075249.188691:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075249.188699:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075249.188725:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075249.188734:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923588224/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075249.188763:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075249.188780:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075249.188782:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075249.188788:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1938 00000100:00080000:3.0:1769075249.188794:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075249.188799:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075249.188804:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923588224:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075249.188811:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075249.188829:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923588352/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075249.188843:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075249.188846:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075249.188848:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075249.188852:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1938 00000100:00080000:3.0:1769075249.188857:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075249.188861:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075249.188864:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075249.188871:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075249.188885:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923588480/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075249.188890:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075249/real 1769075249] req@ffff969e0a7fed80 x1855007923588224/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075249.188903:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075249.188907:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075249.188910:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075249.188914:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923588224/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075249.188917:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1938 00000100:00080000:3.0:1769075249.188936:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075249.188940:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075249.188955:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075249.188961:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075249.188967:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075249.188972:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923588608/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075249.188973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923588224:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075249.188990:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075249.188993:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075249.188996:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075249.189000:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588352:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075249.189004:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1938 00000100:00080000:3.0:1769075249.189010:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075249.189015:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075249.189020:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075249.189028:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075249.189034:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923588736/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075249.189040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075249/real 1769075249] req@ffff969e0a7fe300 x1855007923588352/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075249.189058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923588352/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075249.189064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075249.189068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075249.189071:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588352:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075249.189076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588480:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075249.189087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075249.189094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075249/real 1769075249] req@ffff969e0a7fc380 x1855007923588480/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075249.189101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923588480/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075249.189107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075249.189109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075249.189111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588480:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075249.189121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588608:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075249.189135:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075249.189142:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075249/real 1769075249] req@ffff969e0a7fdf80 x1855007923588608/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075249.189150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923588608/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075249.189164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075249.189166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075249.189168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588608:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075249.189179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588736:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075249.189191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075249.189197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075249/real 1769075249] req@ffff969e0a7fce00 x1855007923588736/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075249.189205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923588736/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075249.189210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075249.189213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075249.189214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588736:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075251.236823:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075253.284797:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075254.308724:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075254.308734:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075254.308740:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075254.308751:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1943 00000100:00080000:0.0:1769075254.308761:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075254.308769:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075254.308804:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923588864/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075254.308845:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075254.308850:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075254.308852:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075254.308857:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1943 00000100:00080000:0.0:1769075254.308862:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075254.308867:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075254.308877:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075254.308893:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923588992/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075254.308909:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075254.308913:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075254.308935:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075254.308937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923588864:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075254.308945:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1943 00000100:00080000:0.0:1769075254.308955:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075254.308961:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075254.308968:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075254.308988:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923589120/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075254.309001:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075254.309005:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075254.309007:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075254.309011:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1943 00000100:00080000:0.0:1769075254.309017:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075254.309022:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075254.309028:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075254.309039:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075254.309045:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923589248/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075254.309059:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075254.309063:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075254.309067:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075254.309074:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1943 00000100:00080000:0.0:1769075254.309079:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075254.309083:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075254.309090:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075254.309103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075254/real 1769075254] req@ffff969e06a60700 x1855007923588864/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075254.309112:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923589376/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075254.309152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923588864/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075254.309166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075254.309172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075254.309177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923588864:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075254.309208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588992:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075254.309237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075254.309251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075254/real 1769075254] req@ffff969e06a60000 x1855007923588992/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075254.309266:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923588992/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075254.309276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075254.309279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075254.309283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923588992:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075254.309290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589120:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075254.309312:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075254.309324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075254/real 1769075254] req@ffff969e06a63800 x1855007923589120/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075254.309337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923589120/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075254.309360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075254.309364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075254.309368:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589120:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075254.309375:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589248:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075254.309399:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075254.309411:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075254/real 1769075254] req@ffff969e06a60a80 x1855007923589248/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075254.309424:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923589248/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075254.309434:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075254.309437:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075254.309441:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589248:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075254.309456:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589376:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075254.309476:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075254.309487:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075254/real 1769075254] req@ffff969e06a61f80 x1855007923589376/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075254.309499:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923589376/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075254.309510:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075254.309513:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075254.309523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589376:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075255.332766:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075257.380736:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075259.428734:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075259.428738:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075259.428745:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075259.428775:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075259.428785:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1948 00000100:00080000:3.0:1769075259.428795:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075259.428800:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075259.428834:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923589504/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075259.428854:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075259.428857:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075259.428858:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075259.428861:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1948 00000100:00080000:3.0:1769075259.428878:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075259.428882:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075259.428889:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075259.428895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923589504:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075259.428903:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923589632/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075259.428912:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075259.428914:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075259.428916:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075259.428919:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1948 00000100:00080000:3.0:1769075259.428921:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075259.428924:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075259.428927:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075259.428935:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923589760/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075259.428941:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075259.428944:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075259.428945:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075259.428947:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1948 00000100:00080000:3.0:1769075259.428957:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075259.428959:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075259.428962:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075259.428970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075259.428971:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923589888/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075259.428989:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075259.428992:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075259.428993:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075259.428996:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1948 00000100:00080000:3.0:1769075259.429000:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075259.429003:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075259.429007:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075259.429014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075259/real 1769075259] req@ffff969e0a7fca80 x1855007923589504/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075259.429017:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923590016/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075259.429061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923589504/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075259.429073:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075259.429077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075259.429082:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923589504:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075259.429107:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589632:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075259.429132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075259.429146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075259/real 1769075259] req@ffff969e0a7ffb80 x1855007923589632/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075259.429156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923589632/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075259.429162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075259.429164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075259.429166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589632:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075259.429171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589760:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075259.429184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075259.429198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075259/real 1769075259] req@ffff969e0a7fd500 x1855007923589760/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075259.429206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923589760/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075259.429212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075259.429214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075259.429216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589760:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075259.429220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589888:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075259.429231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075259.429242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075259/real 1769075259] req@ffff969e0a7fdc00 x1855007923589888/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075259.429249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923589888/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075259.429254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075259.429256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075259.429258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923589888:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075259.429261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590016:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075259.429270:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075259.429276:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075259/real 1769075259] req@ffff969e0a7ff800 x1855007923590016/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075259.429283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923590016/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075259.429293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075259.429296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075259.429298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590016:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075261.476750:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075263.524981:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075264.548828:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075264.548950:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923590144:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769075264.549039:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075264.549084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075264/real 1769075264] req@ffff969e10b63800 x1855007923590144/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075264.549107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b63800 x1855007923590144/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075264.549123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075264.549130:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075264.549139:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923590144:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075264.549169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590272:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075264.549188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075264.549197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075264/real 1769075264] req@ffff969e10b61500 x1855007923590272/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075264.549206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923590272/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075264.549229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075264.549232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075264.549244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590272:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075264.549249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590400:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075264.549267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075264.549275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075264/real 1769075264] req@ffff969e10b61f80 x1855007923590400/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075264.549303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923590400/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075264.549311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075264.549314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075264.549317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590400:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075264.549322:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590528:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075264.549339:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075264.549364:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075264/real 1769075264] req@ffff969e10b60a80 x1855007923590528/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075264.549374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923590528/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075264.549381:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075264.549383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075264.549385:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590528:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075264.549390:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590656:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075264.549403:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075264.549418:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075264/real 1769075264] req@ffff969e10b62a00 x1855007923590656/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075264.549428:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923590656/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075264.549435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075264.549453:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075264.549456:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590656:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075265.572864:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075266.596809:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075267.620906:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075268.644824:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075269.668906:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923590784:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769075269.669003:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075269.669044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075269/real 1769075269] req@ffff969e10b62680 x1855007923590784/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075269.669066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923590784/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075269.669088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075269.669106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075269.669112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923590784:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075269.669144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590912:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075269.669176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075269.669195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075269/real 1769075269] req@ffff969e10b62a00 x1855007923590912/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075269.669208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923590912/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075269.669268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075269.669273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075269.669291:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923590912:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075269.669302:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591040:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769075269.669321:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1769075269.669329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075269.669345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075269/real 1769075269] req@ffff969e10b60a80 x1855007923591040/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075269.669356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923591040/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075269.669365:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075269.669368:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075269.669372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591040:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075269.669378:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591168:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075269.669393:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075269.669402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075269/real 1769075269] req@ffff969e10b61f80 x1855007923591168/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075269.669412:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923591168/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075269.669420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075269.669423:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075269.669434:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591168:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075269.669453:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591296:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075269.669473:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075269.669482:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075269/real 1769075269] req@ffff969e10b61500 x1855007923591296/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075269.669494:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923591296/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075269.669502:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075269.669505:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075269.669508:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591296:192.168.204.143@tcp:8:kworker.0 00010000:00010000:3.0:1769075270.692804:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075271.716787:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075272.740775:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075273.764752:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075274.788727:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075274.788742:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075274.788748:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075274.788760:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1963 00000100:00080000:0.0:1769075274.788775:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075274.788784:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075274.788792:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075274.788843:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923591424/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075274.788874:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075274.788877:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075274.788879:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075274.788884:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1963 00000100:00080000:0.0:1769075274.788889:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075274.788893:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075274.788913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923591424:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769075274.788931:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075274.788956:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923591552/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075274.788968:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075274.788971:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075274.788973:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075274.788979:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1963 00000100:00080000:0.0:1769075274.788983:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075274.788989:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075274.788991:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075274.789005:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075274.789019:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923591680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075274.789030:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075274.789034:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075274.789036:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075274.789042:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075274/real 1769075274] req@ffff969e06a60380 x1855007923591424/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075274.789046:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1963 00000100:00080000:0.0:1769075274.789056:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075274.789071:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075274.789081:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075274.789088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923591424/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075274.789094:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923591808/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075274.789136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075274.789138:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075274.789145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075274.789147:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769075274.789154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923591424:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075274.789159:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075274.789168:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1963 00000100:00080000:0.0:1769075274.789174:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075274.789179:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075274.789185:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075274.789188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591552:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075274.789198:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923591936/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075274.789248:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075274.789260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075274/real 1769075274] req@ffff969e06a62d80 x1855007923591552/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075274.789269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923591552/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075274.789275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075274.789277:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075274.789280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591552:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075274.789285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591680:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075274.789296:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075274.789302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075274/real 1769075274] req@ffff969e06a61500 x1855007923591680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075274.789309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923591680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075274.789314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075274.789316:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075274.789319:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591680:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075274.789326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591808:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075274.789344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075274.789356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075274/real 1769075274] req@ffff969e06a61180 x1855007923591808/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075274.789383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923591808/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075274.789391:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075274.789393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075274.789395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591808:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075274.789410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591936:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075274.789426:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075274.789433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075274/real 1769075274] req@ffff969e06a62680 x1855007923591936/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075274.789440:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923591936/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075274.789446:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075274.789448:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075274.789450:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923591936:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769075275.812825:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075276.836760:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075277.860790:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075278.884816:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075279.908728:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075279.908739:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075279.908744:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075279.908755:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1969 00000100:00080000:0.0:1769075279.908765:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075279.908772:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075279.908815:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075279.908816:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923592064/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075279.908853:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075279.908859:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075279.908863:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075279.908870:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1969 00000100:00080000:0.0:1769075279.908876:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075279.908883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923592064:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075279.908885:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075279.908902:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075279.908940:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923592192/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075279.908955:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075279.908959:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075279.908966:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075279.908969:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075279.908977:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1969 00000100:00080000:0.0:1769075279.908983:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075279.908988:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075279.908994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075279/real 1769075279] req@ffff969e06a61880 x1855007923592064/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075279.908996:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075279.909015:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923592320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075279.909042:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075279.909048:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075279.909050:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075279.909054:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1969 00000100:00080000:0.0:1769075279.909058:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075279.909061:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075279.909064:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075279.909067:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923592064/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075279.909076:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923592448/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075279.909095:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075279.909096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075279.909098:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075279.909104:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075279.909109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075279.909120:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1969 00000100:00100000:2.0:1769075279.909120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923592064:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075279.909127:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075279.909134:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075279.909138:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075279.909147:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923592576/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075279.909161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923592192:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075279.909216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075279.909236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075279/real 1769075279] req@ffff969e06a63100 x1855007923592192/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075279.909253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923592192/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075279.909264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075279.909268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075279.909273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923592192:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075279.909281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923592320:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075279.909302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075279.909313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075279/real 1769075279] req@ffff969e06a62680 x1855007923592320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075279.909328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923592320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075279.909355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075279.909360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075279.909364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923592320:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075279.909372:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923592448:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075279.909397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075279.909455:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075279/real 1769075279] req@ffff969e06a61180 x1855007923592448/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075279.909487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923592448/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075279.909493:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075279.909496:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075279.909498:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923592448:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075279.909512:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923592576:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075279.909528:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075279.909536:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075279/real 1769075279] req@ffff969e06a61500 x1855007923592576/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075279.909544:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923592576/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075279.909550:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075279.909552:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075279.909560:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923592576:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075280.932762:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075281.956777:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075282.980955:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075284.004861:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075285.028682:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075285.028694:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075285.028697:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075285.028706:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1974 00000100:00080000:0.0:1769075285.028715:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075285.028721:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075285.028752:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923592704/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075285.028778:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075285.028781:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075285.028783:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075285.028789:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1974 00000100:00080000:0.0:1769075285.028793:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075285.028797:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075285.028806:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075285.028815:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923592832/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075285.028828:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769075285.028828:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075285.028850:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075285.028852:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075285.028861:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1974 00000100:00080000:0.0:1769075285.028866:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075285.028870:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075285.028874:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075285.028884:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923592960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075285.028891:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075285.028894:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075285.028895:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075285.028897:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1974 00000100:00080000:0.0:1769075285.028900:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075285.028902:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075285.028905:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075285.028914:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923593088/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075285.028921:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075285.028924:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075285.028926:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075285.028939:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1974 00000100:00080000:0.0:1769075285.028942:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075285.028944:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075285.028952:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075285.028963:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923593216/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075286.052827:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075287.076957:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075288.100957:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075289.124779:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075290.148654:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075290.148662:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075290.148666:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075290.148674:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1979 00000100:00080000:0.0:1769075290.148683:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075290.148689:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075290.148719:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075290.148720:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923593344/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075290.148743:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075290.148745:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075290.148747:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075290.148750:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1979 00000100:00080000:0.0:1769075290.148752:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075290.148754:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075290.148761:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075290.148772:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923593472/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075290.148780:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075290.148784:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075290.148786:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075290.148793:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1979 00000100:00080000:0.0:1769075290.148810:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075290.148813:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075290.148818:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075290.148827:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923593600/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075290.148834:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075290.148836:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075290.148837:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075290.148840:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1979 00000100:00080000:0.0:1769075290.148842:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075290.148845:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075290.148848:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075290.148857:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923593728/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075290.148864:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075290.148867:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075290.148869:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075290.148872:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1979 00000100:00080000:0.0:1769075290.148875:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075290.148877:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075290.148880:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075290.148890:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923593856/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075291.172854:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075292.196796:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075293.220893:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075294.244760:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075295.268721:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075295.268731:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075295.268735:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075295.268744:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1984 00000100:00080000:2.0:1769075295.268754:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075295.268761:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075295.268791:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075295.268797:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923593984/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075295.268821:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075295.268824:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075295.268825:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075295.268828:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1984 00000100:00080000:2.0:1769075295.268847:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075295.268852:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075295.268861:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075295.268876:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923594112/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075295.268887:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075295.268890:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075295.268892:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075295.268895:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1984 00000100:00080000:2.0:1769075295.268898:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075295.268900:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075295.268904:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075295.268913:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923594240/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075295.268921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075295.268924:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075295.268925:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075295.268928:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1984 00000100:00080000:2.0:1769075295.268933:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075295.268939:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075295.268943:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075295.268953:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923594368/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075295.268974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075295.268982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075295.268986:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075295.268994:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1984 00000100:00080000:2.0:1769075295.268999:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075295.269002:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075295.269007:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075295.269019:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923594496/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075296.292819:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075297.316773:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075298.340766:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075299.364916:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075300.388755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075300.388768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075300.388774:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075300.388786:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1989 00000100:00080000:2.0:1769075300.388802:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075300.388811:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075300.388869:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923594624/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075300.388900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769075300.388902:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075300.388905:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075300.388912:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075300.388917:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1989 00000100:00080000:2.0:1769075300.388944:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075300.388948:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075300.388959:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075300.388996:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923594752/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075300.389015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075300.389020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075300.389022:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075300.389027:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1989 00000100:00080000:2.0:1769075300.389031:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075300.389035:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075300.389049:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075300.389083:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923594880/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075300.389092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075300.389095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075300.389097:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075300.389101:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1989 00000100:00080000:2.0:1769075300.389117:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075300.389120:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075300.389125:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075300.389135:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923595008/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075300.389156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075300.389160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075300.389163:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075300.389180:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1989 00000100:00080000:2.0:1769075300.389189:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075300.389193:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075300.389203:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075300.389227:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923595136/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075301.412865:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075302.436830:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075303.460745:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075303.461185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075300/real 1769075303] req@ffff969e0a46ca80 x1855007923595136/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075303.461227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923595136/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075303.461242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075303.461248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075303.461252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595136:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075303.461260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075300/real 1769075303] req@ffff969e0a46ce00 x1855007923594752/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075303.461271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923594752/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075303.461277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075303.461304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075303.461307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923594752:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075303.461314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075300/real 1769075303] req@ffff969e0a46e300 x1855007923594880/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075303.461328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923594880/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075303.461339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075303.461344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075303.461347:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923594880:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075303.461356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075300/real 1769075303] req@ffff969e0a46ed80 x1855007923595008/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075303.461383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923595008/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075303.461392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075303.461396:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075303.461400:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595008:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075303.461407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075300/real 1769075303] req@ffff969e0a46d880 x1855007923594624/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075303.461421:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923594624/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075303.461428:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075303.461439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075303.461443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923594624:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075303.780699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075304.484769:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075304.804695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075305.508824:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923595264:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769075305.508881:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075305.508912:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075305/real 1769075305] req@ffff969e10b63800 x1855007923595264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075305.508933:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b63800 x1855007923595264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075305.508963:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075305.508969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075305.508987:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923595264:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075305.509015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595392:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075305.509050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075305.509068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075305/real 1769075305] req@ffff969e10b60700 x1855007923595392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075305.509083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923595392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075305.509092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075305.509096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075305.509101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595392:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075305.509119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595520:192.168.204.143@tcp:38:kworker.0 00010000:00010000:3.0:1769075305.509131:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1769075305.509143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075305.509159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075305/real 1769075305] req@ffff969e10b62d80 x1855007923595520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075305.509172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923595520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075305.509179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075305.509183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075305.509186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595520:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075305.509192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595648:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075305.509211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075305.509223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075305/real 1769075305] req@ffff969e10b60000 x1855007923595648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075305.509235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923595648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075305.509244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075305.509247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075305.509250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595648:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075305.509267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595776:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075305.509288:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075305.509298:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075305/real 1769075305] req@ffff969e10b61180 x1855007923595776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075305.509318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923595776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075305.509326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075305.509330:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075305.509333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923595776:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075305.828682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075306.532724:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075306.852691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075307.556798:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075307.876668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075308.580734:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075308.900714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075309.604694:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075309.924679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075310.628754:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075310.628764:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075310.628768:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075310.628777:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 1999 00010000:00010000:2.0:1769075310.628782:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075310.628787:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075310.628794:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075310.628849:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923595904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075310.628878:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075310.628881:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075310.628883:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075310.628887:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1999 00000100:00080000:0.0:1769075310.628907:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075310.628912:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075310.628923:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075310.628948:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923595904:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075310.628958:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923596032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075310.628973:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075310.628977:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075310.628980:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075310.628984:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1999 00000100:00080000:0.0:1769075310.628989:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075310.628995:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075310.629015:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075310.629016:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075310.629062:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923596160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075310.629072:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075310.629078:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1769075310.629078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075310/real 1769075310] req@ffff969e06a60380 x1855007923595904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075310.629083:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075310.629095:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1999 00000100:00080000:0.0:1769075310.629104:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075310.629110:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075310.629116:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075310.629118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923595904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075310.629129:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923596288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075310.629142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075310.629145:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075310.629151:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075310.629155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075310.629156:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075310.629168:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 1999 00000100:00100000:2.0:1769075310.629171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923595904:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075310.629172:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075310.629181:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075310.629185:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075310.629192:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923596416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075310.629226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596032:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075310.629269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075310.629288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075310/real 1769075310] req@ffff969e06a62d80 x1855007923596032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075310.629304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923596032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075310.629315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075310.629319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075310.629324:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596032:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075310.629332:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596160:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075310.629354:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075310.629367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075310/real 1769075310] req@ffff969e06a61500 x1855007923596160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075310.629381:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923596160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075310.629391:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075310.629411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075310.629415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596160:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075310.629421:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596288:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075310.629439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075310.629451:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075310/real 1769075310] req@ffff969e06a61180 x1855007923596288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075310.629497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923596288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075310.629508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075310.629513:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075310.629518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596288:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075310.629525:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596416:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075310.629550:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075310.629563:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075310/real 1769075310] req@ffff969e06a62680 x1855007923596416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075310.629579:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923596416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075310.629590:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075310.629594:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075310.629598:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596416:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075310.948678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075311.652820:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075311.972707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075312.676813:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075312.996657:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075313.700768:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075314.020687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075314.724779:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075315.044730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075315.748690:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075315.748702:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075315.748727:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075315.748738:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2004 00000100:00080000:0.0:1769075315.748752:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769075315.748754:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075315.748760:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075315.748801:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923596544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075315.748848:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075315.748853:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075315.748855:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075315.748861:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2004 00000100:00080000:0.0:1769075315.748866:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075315.748870:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075315.748880:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075315.748893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923596544:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075315.748897:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923596672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075315.748913:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075315.748916:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075315.748917:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075315.748931:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2004 00000100:00080000:0.0:1769075315.748934:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075315.748937:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075315.748941:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075315.748956:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923596800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075315.748975:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075315.748979:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075315.748982:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075315.748987:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2004 00000800:00000100:2.0:1769075315.748993:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075315.748994:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075315.749005:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075315.749013:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075315.749027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075315/real 1769075315] req@ffff969e06a63100 x1855007923596544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075315.749034:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923596928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075315.749056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923596544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075315.749062:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075315.749067:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075315.749068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075315.749071:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075315.749080:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2004 00000100:00080000:2.0:1769075315.749081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075315.749098:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075315.749098:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923596544:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075315.749110:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075315.749118:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075315.749126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596672:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075315.749141:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923597056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075315.749151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075315.749164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075315/real 1769075315] req@ffff969e06a61880 x1855007923596672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075315.749181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923596672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075315.749186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075315.749190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075315.749192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596672:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075315.749197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596800:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075315.749211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075315.749217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075315/real 1769075315] req@ffff969e06a62680 x1855007923596800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075315.749235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923596800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075315.749241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075315.749244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075315.749246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596800:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075315.749250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596928:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075315.749263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075315.749269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075315/real 1769075315] req@ffff969e06a61180 x1855007923596928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075315.749276:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923596928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075315.749282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075315.749283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075315.749285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923596928:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075315.749296:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597056:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075315.749306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075315.749312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075315/real 1769075315] req@ffff969e06a61500 x1855007923597056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075315.749319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923597056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075315.749333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075315.749335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075315.749338:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597056:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075316.068736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075316.772755:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075317.092692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075317.796754:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075318.116733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075318.820867:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075319.140710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075319.844725:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075320.164720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075320.868818:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075320.868921:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923597184:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769075320.868987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075320.869022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075320/real 1769075320] req@ffff969e10b60380 x1855007923597184/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075320.869052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923597184/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075320.869074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075320.869089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075320.869111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923597184:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075320.869150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597312:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075320.869179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075320.869191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075320/real 1769075320] req@ffff969e10b61180 x1855007923597312/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075320.869210:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923597312/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075320.869217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075320.869220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075320.869222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597312:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075320.869227:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597440:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075320.869243:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075320.869250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075320/real 1769075320] req@ffff969e10b60000 x1855007923597440/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075320.869269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923597440/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075320.869276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075320.869278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075320.869281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597440:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075320.869285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597568:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075320.869298:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075320.869305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075320/real 1769075320] req@ffff969e10b62d80 x1855007923597568/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075320.869313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923597568/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075320.869337:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075320.869340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075320.869343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597568:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075320.869357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597696:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075320.869375:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075320.869387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075320/real 1769075320] req@ffff969e10b60700 x1855007923597696/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075320.869401:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923597696/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075320.869411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075320.869414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075320.869418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923597696:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075321.188668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075321.892839:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075322.212734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075322.916987:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075323.236730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075323.940823:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075324.260693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075324.964874:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075325.284672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075325.988787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075325.988800:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075325.988805:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075325.988816:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2015 00000100:00080000:2.0:1769075325.988829:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075325.988839:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075325.988902:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075325.988919:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923597824/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075325.988989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075325.988993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075325.988996:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075325.989001:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2015 00000100:00080000:2.0:1769075325.989007:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075325.989028:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075325.989039:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075325.989059:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923597952/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075325.989089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075325.989093:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075325.989095:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075325.989103:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2015 00000100:00080000:2.0:1769075325.989109:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075325.989114:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075325.989119:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075325.989135:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923598080/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075325.989145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075325.989149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075325.989152:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075325.989184:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2015 00000100:00080000:2.0:1769075325.989199:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075325.989205:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075325.989212:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075325.989228:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923598208/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075325.989242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075325.989261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075325.989267:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075325.989288:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2015 00000100:00080000:2.0:1769075325.989296:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075325.989302:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075325.989312:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075325.989335:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923598336/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075326.308675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075327.012776:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075327.332685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075328.036788:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075328.356671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075329.060737:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075329.380694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075330.084862:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075330.404696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075331.108787:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075331.108805:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075331.108811:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075331.108825:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2020 00000100:00080000:3.0:1769075331.108839:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769075331.108848:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075331.108850:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075331.108902:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923598464/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075331.108946:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075331.108951:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075331.108954:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075331.108959:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2020 00000100:00080000:3.0:1769075331.108964:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075331.108968:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075331.108980:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075331.108998:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923598592/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075331.109012:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075331.109039:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075331.109049:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075331.109056:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2020 00000100:00080000:3.0:1769075331.109062:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075331.109068:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075331.109075:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075331.109095:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923598720/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075331.109111:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075331.109116:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075331.109118:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075331.109123:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2020 00000100:00080000:3.0:1769075331.109128:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075331.109145:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075331.109157:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075331.109183:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923598848/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075331.109234:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075331.109243:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075331.109252:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075331.109268:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2020 00000100:00080000:3.0:1769075331.109280:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075331.109290:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075331.109314:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075331.109340:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923598976/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075331.428686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075332.132721:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075332.452649:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075333.156737:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075333.476694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075334.180717:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075334.500655:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075335.204740:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075335.524683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00100000:2.0:1769075336.228889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923599104:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769075336.229006:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075336.229058:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075336/real 1769075336] req@ffff969e10b63800 x1855007923599104/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075336.229108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b63800 x1855007923599104/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075336.229137:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075336.229149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075336.229175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923599104:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075336.229214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599232:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075336.229248:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00010000:00010000:3.0:1769075336.229260:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075336.229268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075336/real 1769075336] req@ffff969e10b61500 x1855007923599232/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075336.229282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923599232/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075336.229290:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075336.229294:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075336.229297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599232:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075336.229303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599360:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075336.229341:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075336.229350:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075336/real 1769075336] req@ffff969e10b61f80 x1855007923599360/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075336.229361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923599360/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075336.229369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075336.229383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075336.229386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599360:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075336.229392:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599488:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075336.229409:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075336.229417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075336/real 1769075336] req@ffff969e10b60a80 x1855007923599488/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075336.229428:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923599488/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075336.229436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075336.229439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075336.229442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599488:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075336.229447:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599616:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075336.229502:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075336.229520:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075336/real 1769075336] req@ffff969e10b62a00 x1855007923599616/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075336.229560:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923599616/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075336.229574:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075336.229580:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075336.229585:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599616:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075336.548700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075337.252874:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075337.572684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075338.276751:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075338.596691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075339.300753:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075339.620671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075340.324754:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075340.644681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075341.348694:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075341.348706:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075341.348711:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075341.348722:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2030 00000100:00080000:3.0:1769075341.348733:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075341.348741:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075341.348757:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075341.348780:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923599744/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075341.348803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075341.348806:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075341.348807:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075341.348810:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2030 00000100:00080000:3.0:1769075341.348813:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075341.348817:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075341.348825:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075341.348835:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923599872/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075341.348843:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923599744:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075341.348866:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075341.348868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075341.348870:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075341.348873:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2030 00000100:00080000:3.0:1769075341.348877:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075341.348880:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075341.348883:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075341.348892:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923600000/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075341.348899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075341.348901:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075341.348902:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075341.348904:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2030 00000100:00080000:3.0:1769075341.348907:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075341.348911:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075341.348914:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075341.348919:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075341.348926:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923600128/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075341.348933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075341.348935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075341.348937:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075341.348940:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2030 00000100:00080000:3.0:1769075341.348947:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075341.348952:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075341/real 1769075341] req@ffff969e0a7fca80 x1855007923599744/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075341.348953:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075341.348960:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075341.348971:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923600256/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075341.348984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923599744/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075341.348996:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075341.349001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075341.349005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923599744:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075341.349029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599872:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075341.349061:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075341.349073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075341/real 1769075341] req@ffff969e0a7fce00 x1855007923599872/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075341.349096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923599872/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075341.349104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075341.349108:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075341.349111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923599872:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075341.349117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600000:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075341.349135:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075341.349145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075341/real 1769075341] req@ffff969e0a7fdf80 x1855007923600000/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075341.349156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923600000/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075341.349163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075341.349166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075341.349170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600000:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075341.349176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600128:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075341.349193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075341.349203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075341/real 1769075341] req@ffff969e0a7fc380 x1855007923600128/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075341.349228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923600128/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075341.349236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075341.349239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075341.349242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600128:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075341.349249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600256:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075341.349269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075341.349279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075341/real 1769075341] req@ffff969e0a7fe300 x1855007923600256/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075341.349291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923600256/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075341.349299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075341.349302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075341.349306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600256:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075341.668701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075342.372749:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075342.692664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075343.396749:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075343.716657:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075344.420735:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075344.740688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075345.444740:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075345.764658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075346.468678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075346.468688:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075346.468693:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075346.468704:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2035 00000100:00080000:3.0:1769075346.468715:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075346.468723:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075346.468737:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075346.468759:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923600384/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075346.468791:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075346.468795:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075346.468797:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075346.468803:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2035 00000100:00080000:3.0:1769075346.468808:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075346.468812:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075346.468818:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923600384:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075346.468823:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075346.468838:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923600512/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075346.468868:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075346.468873:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075346.468875:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075346.468879:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2035 00000100:00080000:3.0:1769075346.468884:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075346.468889:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075346.468892:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075346.468899:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075346.468915:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923600640/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075346.468924:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075346/real 1769075346] req@ffff969e0a7fed80 x1855007923600384/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075346.468939:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923600384/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075346.468943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075346.468951:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075346.468951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075346.468956:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075346.468959:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075346.468966:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923600384:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075346.468966:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2035 00000100:00080000:3.0:1769075346.468974:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075346.468981:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075346.468986:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075346.468990:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600512:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075346.469002:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923600768/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075346.469011:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075346.469022:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075346.469023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075346/real 1769075346] req@ffff969e0a7fe680 x1855007923600512/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075346.469030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075346.469034:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075346.469043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923600512/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075346.469043:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2035 00000100:00080000:2.0:1769075346.469051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075346.469056:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075346.469056:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075346.469062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600512:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075346.469067:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075346.469069:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600640:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075346.469077:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075346.469087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075346.469096:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923600896/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075346.469098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075346/real 1769075346] req@ffff969e0a7fc700 x1855007923600640/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075346.469139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923600640/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075346.469145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075346.469147:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075346.469149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600640:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075346.469159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600768:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075346.469172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075346.469179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075346/real 1769075346] req@ffff969e0a7fd180 x1855007923600768/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075346.469186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923600768/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075346.469191:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075346.469193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075346.469195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600768:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075346.469198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600896:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075346.469207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075346.469212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075346/real 1769075346] req@ffff969e0a7fea00 x1855007923600896/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075346.469219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923600896/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075346.469233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075346.469235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075346.469238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923600896:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075346.788665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075347.492731:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075347.812643:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075348.516739:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075348.836655:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075349.540775:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075349.860667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075350.564727:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075350.884695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075351.588723:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075351.588736:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075351.588741:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075351.588751:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2040 00000100:00080000:3.0:1769075351.588765:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075351.588772:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075351.588820:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923601024/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075351.588826:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075351.588842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075351.588846:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075351.588848:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075351.588851:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2040 00000100:00100000:2.0:1769075351.588868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923601024:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075351.588878:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075351.588882:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075351.588894:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075351.588909:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923601152/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075351.588920:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075351.588947:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075351.588950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075351.588953:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075351.588959:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2040 00000100:00080000:3.0:1769075351.588965:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075351.588969:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075351.588974:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075351.588985:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923601280/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075351.588995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075351.588999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075351/real 1769075351] req@ffff969e0a7fd880 x1855007923601024/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075351.589001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075351.589006:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075351.589024:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2040 00000100:00080000:3.0:1769075351.589033:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075351.589038:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923601024/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075351.589043:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769075351.589063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075351.589063:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075351.589070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075351.589076:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923601024:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075351.589087:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923601408/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075351.589101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075351.589102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601152:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075351.589118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075351.589122:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075351.589127:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2040 00000100:00080000:3.0:1769075351.589136:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075351.589144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075351.589147:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075351.589162:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075351.589186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075351/real 1769075351] req@ffff969e0a7fea00 x1855007923601152/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075351.589186:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923601536/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075351.589245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923601152/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075351.589253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075351.589256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075351.589259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601152:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075351.589265:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601280:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075351.589282:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075351.589289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075351/real 1769075351] req@ffff969e0a7fd180 x1855007923601280/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075351.589297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923601280/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075351.589302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075351.589304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075351.589306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601280:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075351.589316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601408:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075351.589327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075351.589334:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075351/real 1769075351] req@ffff969e0a7fc700 x1855007923601408/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075351.589341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923601408/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075351.589352:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075351.589355:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075351.589357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601408:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075351.589361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601536:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075351.589374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075351.589380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075351/real 1769075351] req@ffff969e0a7fe680 x1855007923601536/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075351.589388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923601536/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075351.589393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075351.589395:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075351.589397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601536:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075351.908659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075352.612735:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075352.932694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075353.636789:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075353.956651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075354.660723:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075354.980686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075355.684694:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075356.004712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075356.708665:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075356.708676:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075356.708681:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075356.708692:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2045 00000100:00080000:3.0:1769075356.708702:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075356.708709:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075356.708733:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075356.708747:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923601664/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075356.708776:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075356.708779:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075356.708781:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075356.708787:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2045 00000100:00080000:3.0:1769075356.708793:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075356.708797:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075356.708803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923601664:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075356.708810:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075356.708836:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923601792/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075356.708850:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075356.708853:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075356.708855:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075356.708859:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2045 00000100:00080000:3.0:1769075356.708872:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075356.708878:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075356.708884:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075356.708885:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075356.708902:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923601920/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075356.708914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075356.708918:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075356.708920:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075356.708925:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2045 00000100:00100000:2.0:1769075356.708928:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075356/real 1769075356] req@ffff969e0a7fed80 x1855007923601664/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075356.708931:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075356.708940:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075356.708947:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075356.708948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923601664/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075356.708959:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075356.708964:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075356.708964:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923602048/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075356.708972:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923601664:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075356.708986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075356.708990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075356.708992:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075356.708998:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2045 00000100:00100000:2.0:1769075356.709000:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601792:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075356.709004:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075356.709012:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075356.709018:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075356.709021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075356.709039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075356/real 1769075356] req@ffff969e0a7fe300 x1855007923601792/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075356.709040:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923602176/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075356.709062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923601792/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075356.709068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075356.709074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075356.709079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601792:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075356.709100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601920:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075356.709140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075356.709153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075356/real 1769075356] req@ffff969e0a7fc380 x1855007923601920/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075356.709177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923601920/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075356.709185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075356.709188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075356.709191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923601920:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075356.709198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602048:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075356.709218:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075356.709228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075356/real 1769075356] req@ffff969e0a7fdf80 x1855007923602048/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075356.709239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923602048/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075356.709246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075356.709249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075356.709252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602048:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075356.709266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602176:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075356.709294:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075356.709305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075356/real 1769075356] req@ffff969e0a7fce00 x1855007923602176/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075356.709317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923602176/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075356.709337:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075356.709340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075356.709343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602176:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075357.028692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075357.732731:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075358.052716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075358.756743:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075359.076685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075359.780773:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075360.100667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075360.804726:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075361.124711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075361.828648:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075361.828663:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075361.828669:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075361.828682:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2051 00000100:00080000:3.0:1769075361.828697:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075361.828706:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075361.828748:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075361.828760:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923602304/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075361.828795:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075361.828800:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075361.828803:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075361.828810:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2051 00000100:00080000:3.0:1769075361.828816:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075361.828822:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075361.828829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923602304:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075361.828840:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075361.828864:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923602432/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075361.828881:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075361.828886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075361.828891:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075361.828896:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2051 00000800:00000100:2.0:1769075361.828902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075361.828905:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075361.828913:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075361.828938:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075361/real 1769075361] req@ffff969e0a7fca80 x1855007923602304/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075361.828948:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075361.828956:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923602304/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075361.828967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769075361.828973:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923602560/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075361.828975:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075361.828984:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923602304:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075361.829012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075361.829016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075361.829019:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075361.829020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602432:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075361.829045:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2051 00000100:00080000:3.0:1769075361.829054:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075361.829061:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075361.829064:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075361.829071:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075361.829092:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923602688/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075361.829098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075361/real 1769075361] req@ffff969e0a7ffb80 x1855007923602432/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075361.829137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075361.829139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923602432/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075361.829149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075361.829156:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075361.829167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075361.829171:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2051 00000100:00080000:2.0:1769075361.829178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075361.829181:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075361.829186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602432:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075361.829192:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075361.829203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602560:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075361.829203:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075361.829232:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923602816/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075361.829244:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075361.829268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075361/real 1769075361] req@ffff969e0a7fd500 x1855007923602560/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075361.829278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923602560/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075361.829283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075361.829286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075361.829288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602560:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075361.829301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602688:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075361.829312:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075361.829318:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075361/real 1769075361] req@ffff969e0a7fdc00 x1855007923602688/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075361.829325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923602688/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075361.829336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075361.829339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075361.829341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602688:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075361.829345:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602816:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075361.829358:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075361.829365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075361/real 1769075361] req@ffff969e0a7ff800 x1855007923602816/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075361.829372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923602816/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075361.829377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075361.829379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075361.829380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923602816:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075362.148708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075362.852759:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075363.172700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075363.876772:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075364.196650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075364.900739:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075365.220693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075365.924777:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075366.244709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075366.948674:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075366.948686:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075366.948713:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075366.948730:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2056 00010000:00010000:2.0:1769075366.948741:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075366.948741:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075366.948750:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075366.948779:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923602944/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075366.948801:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075366.948804:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075366.948805:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075366.948808:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2056 00000100:00080000:3.0:1769075366.948811:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075366.948813:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075366.948821:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075366.948832:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923603072/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075366.948841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923602944:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075366.948843:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075366.948849:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075366.948851:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075366.948854:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2056 00000100:00080000:3.0:1769075366.948858:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075366.948863:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075366.948866:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075366.948874:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923603200/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075366.948892:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075366.948895:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075366.948896:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075366.948899:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2056 00000100:00080000:3.0:1769075366.948905:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075366.948908:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075366.948912:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075366.948921:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923603328/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075366.948928:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075366.948931:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075366.948933:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075366.948936:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2056 00000100:00080000:3.0:1769075366.948938:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075366.948941:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075366.948959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923603072:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075366.948959:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075366.948973:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923603456/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075366.948998:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923603200:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075366.949019:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923603328:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075366.949040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923603456:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075367.268684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075367.972800:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075368.292659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075368.996781:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075369.316671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075370.020878:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769075370.020880:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769075370.020943:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769075370.020963:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075370.021011:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075370.021022:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075370.021030:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075370.021038:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075370.340678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075371.044805:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075371.364663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075372.068655:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075372.068669:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075372.068673:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075372.068698:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2061 00000100:00080000:3.0:1769075372.068723:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075372.068730:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075372.068761:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923603584/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075372.068785:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075372.068788:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075372.068790:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075372.068812:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2061 00000100:00080000:3.0:1769075372.068816:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075372.068819:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075372.068825:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:3.0:1769075372.068828:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075372.068843:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923603712/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075372.068872:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075372.068876:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075372.068878:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075372.068882:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2061 00000100:00080000:3.0:1769075372.068886:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075372.068890:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075372.068896:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075372.068913:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923603840/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075372.068929:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075372.068943:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075372.068947:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075372.068954:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2061 00000100:00080000:3.0:1769075372.068963:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075372.068969:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075372.068976:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075372.068994:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923603968/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075372.069009:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075372.069017:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075372.069023:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075372.069031:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2061 00000100:00080000:3.0:1769075372.069038:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075372.069049:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075372.069059:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075372.069078:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923604096/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075372.388678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075373.092884:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075373.412658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075374.116700:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075374.436705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075375.140799:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075375.460714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075376.164698:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075376.484733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075377.188682:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075377.188694:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075377.188700:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075377.188712:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2066 00000100:00080000:3.0:1769075377.188725:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075377.188733:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075377.188756:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075377.188795:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923604224/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075377.188841:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075377.188844:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075377.188845:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075377.188849:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2066 00000100:00080000:3.0:1769075377.188853:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075377.188858:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075377.188869:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075377.188890:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923604352/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075377.188900:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075377.188903:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075377.188905:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075377.188908:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2066 00000100:00080000:3.0:1769075377.188911:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075377.188913:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075377.188918:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075377.188927:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923604480/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075377.188933:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075377.188936:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075377.188937:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075377.188940:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2066 00000100:00080000:3.0:1769075377.188948:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075377.188952:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075377.188956:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075377.188967:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923604608/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075377.188975:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075377.188978:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075377.188979:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075377.188982:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2066 00000100:00080000:3.0:1769075377.188984:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075377.188986:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075377.188990:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075377.188997:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923604736/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075377.508713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075378.212736:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075378.532729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075379.236827:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075379.556746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075380.260770:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075380.580707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075381.284793:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075381.604787:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075382.308667:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075382.308678:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075382.308683:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075382.308735:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2071 00000100:00080000:3.0:1769075382.308745:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075382.308751:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075382.308793:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923604864/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075382.308814:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075382.308829:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075382.308831:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075382.308835:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2071 00000100:00080000:3.0:1769075382.308839:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075382.308844:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075382.308853:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075382.308869:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923604992/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075382.308877:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075382.308879:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075382.308888:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075382.308889:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075382.308892:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2071 00000100:00080000:3.0:1769075382.308898:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075382.308902:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075382.308907:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075382.308919:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923605120/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075382.308926:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075382.308929:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075382.308930:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075382.308933:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2071 00000100:00080000:3.0:1769075382.308936:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075382.308939:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075382.308950:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075382.308962:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923605248/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075382.308987:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075382.308994:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075382.308999:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075382.309008:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2071 00000100:00080000:3.0:1769075382.309014:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075382.309019:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075382.309025:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075382.309063:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923605376/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075382.628761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075383.332782:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075383.652731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075384.356814:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075384.676738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075385.380803:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075385.700720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075386.404744:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075386.724691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075387.428676:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075387.428687:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075387.428692:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075387.428704:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2076 00000100:00080000:3.0:1769075387.428715:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075387.428723:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075387.428734:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075387.428767:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923605504/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075387.428790:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075387.428792:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075387.428794:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075387.428797:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2076 00000100:00080000:3.0:1769075387.428816:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075387.428822:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075387.428833:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075387.428856:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923605632/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075387.428870:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075387.428873:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075387.428875:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075387.428880:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2076 00000100:00080000:3.0:1769075387.428885:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075387.428892:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075387.428900:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075387.428913:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923605760/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075387.428922:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075387.428924:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075387.428926:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075387.428931:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2076 00000100:00080000:3.0:1769075387.428938:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075387.428945:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075387.428950:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075387.428963:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923605888/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075387.428997:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075387.429005:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075387.429010:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075387.429016:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2076 00000100:00080000:3.0:1769075387.429021:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075387.429026:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075387.429035:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075387.429049:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923606016/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075387.748659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075388.452777:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075388.772693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075389.476777:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075389.796705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075390.500775:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075390.820640:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075391.524788:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075391.844751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075392.548816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075392.548825:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075392.548829:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075392.548838:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2081 00010000:00010000:3.0:1769075392.548839:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075392.548848:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075392.548855:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075392.548920:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923606144/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075392.548962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075392.548967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075392.548969:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075392.548975:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2081 00000100:00080000:2.0:1769075392.548980:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075392.548985:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075392.548996:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075392.549015:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923606272/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075392.549047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075392.549051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075392.549053:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075392.549076:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2081 00000100:00080000:2.0:1769075392.549079:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075392.549083:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075392.549104:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075392.549116:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923606400/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075392.549124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075392.549145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075392.549147:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075392.549150:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2081 00000100:00080000:2.0:1769075392.549155:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075392.549159:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075392.549176:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075392.549189:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923606528/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075392.549203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075392.549207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075392.549209:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075392.549212:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2081 00000100:00080000:2.0:1769075392.549217:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075392.549221:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075392.549225:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075392.549235:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923606656/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075392.868682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075393.572796:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075393.892738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075394.596861:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075394.916686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075395.620793:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075395.940696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075396.644831:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075396.964782:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075397.668745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075397.668761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075397.668768:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075397.668782:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2086 00010000:00010000:2.0:1769075397.668793:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075397.668798:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075397.668810:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075397.668878:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923606784/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075397.668914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075397.668919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075397.668922:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075397.668949:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2086 00000100:00080000:3.0:1769075397.668959:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075397.668966:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075397.668978:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075397.668997:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923606912/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075397.669012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075397.669016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075397.669019:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075397.669024:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2086 00000100:00080000:3.0:1769075397.669044:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075397.669048:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075397.669054:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075397.669069:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923607040/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075397.669080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075397.669085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075397.669087:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075397.669092:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2086 00000100:00080000:3.0:1769075397.669098:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075397.669107:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075397.669114:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075397.669128:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923607168/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075397.669154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075397.669162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075397.669167:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075397.669173:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2086 00000100:00080000:3.0:1769075397.669177:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075397.669183:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075397.669190:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075397.669208:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923607296/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075397.988690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075398.692788:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075399.012684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075399.716744:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075400.036722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075400.740719:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075401.060662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075401.764832:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075402.084688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075402.788691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075402.788701:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075402.788705:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075402.788715:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2091 00000100:00080000:3.0:1769075402.788726:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075402.788733:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075402.788769:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923607424/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075402.788780:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075402.788804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075402.788810:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075402.788813:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075402.788818:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2091 00000100:00080000:3.0:1769075402.788824:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075402.788831:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075402.788859:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075402.788878:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923607552/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075402.788890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075402.788893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075402.788894:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075402.788897:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2091 00000100:00080000:3.0:1769075402.788911:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075402.788918:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075402.788925:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075402.788937:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923607680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075402.788945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075402.788948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075402.788950:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075402.788957:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2091 00000100:00080000:3.0:1769075402.788961:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075402.788963:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075402.788971:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075402.788983:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923607808/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075402.788993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075402.788999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075402.789009:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075402.789022:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2091 00000100:00080000:3.0:1769075402.789031:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075402.789038:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075402.789044:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075402.789067:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923607936/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075403.108691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075403.812715:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075404.132693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075404.836805:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075405.156746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075405.860784:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075406.180686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075406.884807:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075407.204716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075407.908694:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075407.908707:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075407.908712:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075407.908725:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2097 00000100:00080000:0.0:1769075407.908766:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769075407.908767:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075407.908775:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075407.908819:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923608064/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075407.908845:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075407.908849:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075407.908851:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075407.908856:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2097 00000100:00080000:0.0:1769075407.908861:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075407.908866:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075407.908877:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075407.908891:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923608192/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075407.908901:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075407.908905:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075407.908906:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075407.908910:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2097 00000100:00080000:0.0:1769075407.908914:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075407.908917:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075407.908922:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075407.908930:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923608320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075407.908939:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075407.908949:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075407.908951:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075407.908955:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2097 00000100:00080000:0.0:1769075407.908959:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075407.908962:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075407.908967:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075407.908977:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923608448/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075407.909003:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075407.909006:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075407.909008:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075407.909012:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2097 00000100:00080000:0.0:1769075407.909015:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075407.909019:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075407.909024:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075407.909034:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923608576/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075408.228674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075408.932786:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075409.252750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075409.956745:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075410.276702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075410.980753:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075411.300737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075412.004765:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075412.324706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075413.028705:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075413.028718:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075413.028723:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075413.028737:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2102 00010000:00010000:2.0:1769075413.028751:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075413.028752:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075413.028763:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075413.028812:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923608704/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075413.028840:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075413.028843:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075413.028844:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075413.028849:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2102 00000100:00080000:3.0:1769075413.028851:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075413.028855:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075413.028866:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075413.028876:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923608832/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075413.028912:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075413.028915:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075413.028916:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075413.028921:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2102 00000100:00080000:3.0:1769075413.028924:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075413.028927:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075413.028931:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075413.028941:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923608960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075413.028956:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075413.028963:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075413.028964:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075413.028967:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2102 00000100:00080000:3.0:1769075413.028971:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075413.028979:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075413.028985:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075413.028994:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923609088/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075413.029004:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075413.029008:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075413.029010:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075413.029015:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2102 00000100:00080000:3.0:1769075413.029029:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075413.029038:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075413.029045:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075413.029066:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923609216/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075413.348773:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075414.052878:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075414.372710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075415.076787:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075415.396738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075416.100880:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075416.420739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075417.124815:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075417.444769:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075418.148769:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075418.148783:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075418.148789:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075418.148802:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2107 00000100:00080000:3.0:1769075418.148817:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075418.148827:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075418.148865:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075418.148879:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923609344/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075418.148918:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075418.148923:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075418.148925:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075418.148930:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2107 00000100:00080000:3.0:1769075418.148935:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075418.148940:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075418.148952:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075418.148968:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923609472/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075418.148988:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075418.149022:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075418.149025:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075418.149033:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2107 00000100:00080000:3.0:1769075418.149041:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075418.149048:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075418.149055:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075418.149075:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923609600/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075418.149134:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075418.149142:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075418.149146:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075418.149155:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2107 00000100:00080000:3.0:1769075418.149164:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075418.149171:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075418.149183:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075418.149217:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923609728/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075418.149284:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075418.149303:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075418.149314:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075418.149325:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2107 00000100:00080000:3.0:1769075418.149337:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075418.149353:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075418.149382:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075418.149418:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923609856/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075418.468758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075419.172857:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075419.492698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075420.196757:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075420.516745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075421.220799:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075421.540691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075422.244802:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075422.564689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075423.268766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075423.268780:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075423.268786:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075423.268798:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2112 00000100:00080000:3.0:1769075423.268810:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075423.268819:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075423.268852:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075423.268866:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923609984/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075423.268896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075423.268899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075423.268901:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075423.268904:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2112 00000100:00080000:3.0:1769075423.268907:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075423.268910:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075423.268921:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075423.268935:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923610112/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075423.268949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075423.268953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075423.268958:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075423.268968:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2112 00000100:00080000:3.0:1769075423.268996:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075423.269015:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075423.269020:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075423.269034:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923610240/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075423.269054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075423.269056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075423.269059:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075423.269064:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2112 00000100:00080000:3.0:1769075423.269069:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075423.269074:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075423.269088:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075423.269112:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923610368/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075423.269125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075423.269132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075423.269136:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075423.269146:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2112 00000100:00080000:3.0:1769075423.269156:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075423.269164:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075423.269170:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075423.269184:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923610496/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075423.588680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075424.292899:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075424.612733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075425.316789:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075425.636694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075426.340934:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075426.660734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075427.364887:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075427.684755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075428.388766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075428.388784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075428.388792:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075428.388822:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2117 00000100:00080000:3.0:1769075428.388841:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075428.388854:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075428.388946:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923610624/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075428.388956:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075428.388992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075428.388998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075428.389001:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075428.389007:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2117 00000100:00080000:3.0:1769075428.389015:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075428.389022:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075428.389036:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075428.389066:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923610752/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075428.389095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075428.389102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075428.389106:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075428.389114:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2117 00000100:00080000:3.0:1769075428.389123:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075428.389132:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075428.389170:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075428.389205:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923610880/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075428.389230:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075428.389238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075428.389243:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075428.389257:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2117 00000100:00080000:3.0:1769075428.389270:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075428.389305:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075428.389315:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075428.389337:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923611008/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075428.389396:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075428.389404:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075428.389407:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075428.389415:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2117 00000100:00080000:3.0:1769075428.389421:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075428.389426:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075428.389435:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075428.389465:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923611136/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075428.708775:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075429.412840:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075429.732659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075430.436731:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075430.756727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075431.460855:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075431.780659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075432.484827:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075432.804745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075433.508804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075433.508821:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075433.508853:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075433.508887:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2122 00000100:00080000:2.0:1769075433.508903:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769075433.508910:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075433.508913:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075433.508999:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923611264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075433.509024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075433.509027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075433.509029:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075433.509032:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2122 00000100:00080000:2.0:1769075433.509035:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075433.509039:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075433.509073:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075433.509093:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923611392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075433.509133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075433.509137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075433.509139:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075433.509145:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2122 00000100:00080000:2.0:1769075433.509150:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075433.509171:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075433.509175:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075433.509184:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c000 x1855007923611520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075433.509192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075433.509195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075433.509196:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075433.509198:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2122 00000100:00080000:2.0:1769075433.509201:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075433.509204:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075433.509207:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075433.509215:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c700 x1855007923611648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075433.509222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075433.509225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075433.509227:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075433.509229:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2122 00000100:00080000:2.0:1769075433.509247:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075433.509253:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075433.509260:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075433.509275:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923611776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075433.828760:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075434.532777:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075434.852653:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075435.556880:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075435.876761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075436.580816:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769075436.580890:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769075436.580935:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769075436.580955:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075436.580998:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075436.581008:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075436.581033:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075436.581040:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075436.900696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075437.604929:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075437.924812:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075438.628778:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075438.628804:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075438.628818:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075438.628832:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2127 00000100:00080000:3.0:1769075438.628845:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075438.628852:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075438.628896:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923611904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075438.628930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075438.628934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:2.0:1769075438.628936:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075438.628937:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075438.628946:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2127 00000100:00080000:3.0:1769075438.628951:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075438.628957:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075438.628969:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075438.628982:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923612032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075438.628998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075438.629001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075438.629002:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075438.629004:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2127 00000100:00080000:3.0:1769075438.629006:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075438.629009:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075438.629045:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075438.629055:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923612160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075438.629063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075438.629066:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075438.629067:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075438.629069:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2127 00000100:00080000:3.0:1769075438.629073:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075438.629077:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075438.629084:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075438.629095:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923612288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075438.629102:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075438.629104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075438.629105:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075438.629108:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2127 00000100:00080000:3.0:1769075438.629112:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075438.629114:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075438.629118:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075438.629127:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923612416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075438.948765:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075439.653031:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075439.972694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075440.676811:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075440.996680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075441.700914:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075442.020762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075442.724904:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075443.044691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075443.748708:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075443.748721:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075443.748726:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075443.748736:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2132 00000100:00080000:3.0:1769075443.748747:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075443.748754:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075443.748783:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075443.748792:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923612544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075443.748826:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075443.748831:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075443.748834:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075443.748841:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2132 00000100:00080000:3.0:1769075443.748847:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075443.748853:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075443.748865:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075443.748891:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923612672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075443.748909:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075443.748914:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075443.748916:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075443.748922:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2132 00000100:00080000:3.0:1769075443.748928:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075443.748939:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075443.748948:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075443.749003:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923612800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075443.749050:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075443.749061:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075443.749068:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075443.749075:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2132 00000100:00080000:3.0:1769075443.749080:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075443.749085:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075443.749093:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075443.749111:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923612928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075443.749131:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075443.749135:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075443.749137:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075443.749145:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2132 00000100:00080000:3.0:1769075443.749152:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075443.749159:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075443.749165:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075443.749177:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923613056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075444.068770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075444.773002:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075445.092819:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075445.796801:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075446.116777:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075446.820835:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075447.140830:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075447.844716:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075448.164676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075448.868694:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075448.868709:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075448.868714:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075448.868751:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2138 00000100:00080000:3.0:1769075448.868763:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769075448.868764:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075448.868771:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075448.868813:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923613184/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075448.868839:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075448.868841:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075448.868843:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075448.868846:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2138 00000100:00080000:3.0:1769075448.868849:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075448.868851:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075448.868861:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075448.868871:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923613312/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075448.868879:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075448.868883:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075448.868886:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075448.868893:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2138 00000100:00080000:3.0:1769075448.868896:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075448.868899:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075448.868901:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075448.868910:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923613440/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075448.868926:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075448.868930:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075448.868931:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075448.868935:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2138 00000100:00080000:3.0:1769075448.868938:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075448.868940:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075448.868944:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075448.868953:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923613568/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075448.868959:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075448.868963:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075448.868965:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075448.868971:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2138 00000100:00080000:3.0:1769075448.868973:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075448.868976:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075448.868979:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075448.868987:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923613696/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075449.188686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075449.892739:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075450.212747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075450.916726:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075451.236666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075451.940754:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075452.260731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075452.964879:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075453.284702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075453.988686:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075453.988695:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075453.988700:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075453.988710:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2143 00000100:00080000:3.0:1769075453.988739:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769075453.988743:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075453.988748:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075453.988777:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923613824/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075453.988798:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075453.988802:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075453.988804:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075453.988808:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2143 00000100:00080000:3.0:1769075453.988812:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075453.988817:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075453.988842:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075453.988855:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923613952/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075453.988866:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075453.988869:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075453.988871:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075453.988875:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2143 00000100:00080000:3.0:1769075453.988878:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075453.988882:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075453.988886:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075453.988895:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923614080/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075453.988904:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075453.988913:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075453.988915:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075453.988919:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2143 00000100:00080000:3.0:1769075453.988923:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075453.988927:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075453.988931:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075453.988942:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923614208/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075453.988951:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075453.988955:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075453.988957:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075453.988961:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2143 00000100:00080000:3.0:1769075453.988965:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075453.988968:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075453.988988:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075453.988996:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923614336/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075454.308687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075455.012848:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075455.332660:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075456.036775:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075456.356698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075457.060854:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075457.380751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075458.084849:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075458.404750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075459.108811:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075459.108825:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075459.108832:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075459.108848:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2148 00000100:00080000:3.0:1769075459.108865:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075459.108877:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075459.108929:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923614464/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075459.108951:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075459.108999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075459.109004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075459.109007:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075459.109013:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2148 00000100:00080000:3.0:1769075459.109018:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075459.109023:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075459.109036:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075459.109057:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923614592/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075459.109073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075459.109081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075459.109090:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075459.109101:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2148 00000100:00080000:3.0:1769075459.109120:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075459.109129:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075459.109135:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075459.109156:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923614720/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075459.109168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075459.109173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075459.109175:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075459.109180:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2148 00000100:00080000:3.0:1769075459.109196:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075459.109202:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075459.109210:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075459.109225:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923614848/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075459.109241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075459.109253:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075459.109258:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075459.109266:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2148 00000100:00080000:3.0:1769075459.109275:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075459.109283:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075459.109291:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075459.109308:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923614976/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075459.428691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075460.132849:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075460.452724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075461.156903:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075461.476673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075462.180797:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075462.500682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075463.204821:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075463.524714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075464.228735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075464.228753:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075464.228761:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075464.228778:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2153 00000100:00080000:3.0:1769075464.228795:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075464.228807:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075464.228865:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923615104/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075464.228916:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075464.228918:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075464.228974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075464.228979:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075464.228988:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2153 00000100:00080000:3.0:1769075464.228997:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075464.229006:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075464.229020:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075464.229056:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923615232/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075464.229077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075464.229083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075464.229086:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075464.229093:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2153 00000100:00080000:3.0:1769075464.229099:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075464.229106:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075464.229114:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075464.229133:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923615360/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075464.229151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075464.229158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075464.229162:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075464.229169:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2153 00000100:00080000:3.0:1769075464.229191:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075464.229199:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075464.229225:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075464.229245:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923615488/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075464.229263:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075464.229270:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075464.229273:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075464.229280:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2153 00000100:00080000:3.0:1769075464.229288:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075464.229295:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075464.229304:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075464.229379:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923615616/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075464.548755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075465.252852:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075465.572703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075466.276877:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075466.596668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075467.300827:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075467.620762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075468.324864:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075468.644693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075469.348731:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075469.348748:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075469.348755:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075469.348768:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2158 00010000:00010000:3.0:1769075469.348777:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075469.348783:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075469.348792:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075469.348838:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923615744/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075469.348884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075469.348888:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075469.348890:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075469.348894:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2158 00000100:00080000:2.0:1769075469.348898:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075469.348903:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075469.348918:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075469.348941:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923615872/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075469.348961:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075469.348990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075469.348993:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075469.349000:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2158 00000100:00080000:2.0:1769075469.349025:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075469.349032:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075469.349041:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075469.349079:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923616000/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075469.349089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075469.349092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075469.349094:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075469.349098:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2158 00000100:00080000:2.0:1769075469.349102:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075469.349116:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075469.349124:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075469.349138:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923616128/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075469.349146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075469.349151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075469.349155:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075469.349164:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2158 00000100:00080000:2.0:1769075469.349169:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075469.349174:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075469.349189:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075469.349203:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923616256/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075469.668733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075470.372813:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075470.692696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075471.396803:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075471.716677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075472.420909:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075472.740716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075473.444810:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075473.764723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075474.468795:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075474.468810:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075474.468816:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075474.468829:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2163 00000100:00080000:3.0:1769075474.468843:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075474.468861:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075474.468925:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075474.468931:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923616384/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075474.468969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075474.468994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075474.468996:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075474.469001:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2163 00000100:00080000:3.0:1769075474.469004:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075474.469009:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075474.469036:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075474.469055:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923616512/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075474.469086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075474.469091:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075474.469094:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075474.469101:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2163 00000100:00080000:3.0:1769075474.469106:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075474.469112:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075474.469119:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075474.469132:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923616640/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075474.469157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075474.469161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075474.469163:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075474.469200:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2163 00000100:00080000:3.0:1769075474.469204:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075474.469207:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075474.469222:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075474.469247:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923616768/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075474.469255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075474.469258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075474.469260:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075474.469265:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2163 00000100:00080000:3.0:1769075474.469270:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075474.469274:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075474.469280:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075474.469295:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923616896/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075474.788693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075475.492857:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075475.812724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075476.516900:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075476.836764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075477.540910:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075477.860691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075478.564749:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075478.884711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075479.588689:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075479.588704:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075479.588718:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075479.588734:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2168 00000100:00080000:3.0:1769075479.588750:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075479.588759:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075479.588827:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923617024/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075479.588833:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075479.588864:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075479.588868:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075479.588890:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075479.588895:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2168 00000100:00080000:3.0:1769075479.588898:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075479.588902:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075479.588933:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075479.588954:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923617152/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075479.588966:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075479.588969:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075479.588972:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075479.588976:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2168 00000100:00080000:3.0:1769075479.588980:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075479.588983:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075479.588988:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075479.588998:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923617280/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075479.589008:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075479.589012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075479.589014:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075479.589017:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2168 00000100:00080000:3.0:1769075479.589021:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075479.589029:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075479.589037:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075479.589116:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923617408/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075479.589146:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075479.589149:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075479.589150:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075479.589154:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2168 00000100:00080000:3.0:1769075479.589159:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075479.589166:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075479.589175:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075479.589188:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923617536/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075479.908824:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075480.612864:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075480.932678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075481.636774:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075481.956688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075482.660830:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075482.980724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075483.684768:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075484.004661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075484.708748:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075484.708759:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075484.708764:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075484.708774:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2173 00000100:00080000:0.0:1769075484.708786:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075484.708794:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075484.708820:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075484.708832:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923617664/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075484.708860:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075484.708864:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075484.708865:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075484.708869:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2173 00000100:00080000:0.0:1769075484.708872:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075484.708876:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075484.708885:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075484.708901:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923617792/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075484.708936:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075484.708945:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075484.708959:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075484.708966:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2173 00000100:00080000:0.0:1769075484.708988:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075484.708994:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075484.709001:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075484.709032:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923617920/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075484.709045:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075484.709049:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075484.709051:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075484.709055:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2173 00000100:00080000:0.0:1769075484.709061:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075484.709064:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075484.709069:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075484.709079:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923618048/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075484.709088:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075484.709091:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075484.709095:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075484.709102:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2173 00000100:00080000:0.0:1769075484.709106:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075484.709112:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075484.709118:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075484.709129:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923618176/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075485.028707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075485.732817:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075486.052684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075486.756755:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075487.076688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075487.780729:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075488.100709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075488.804811:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075489.124696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075489.828720:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075489.828730:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075489.828735:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075489.828745:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2179 00000100:00080000:2.0:1769075489.828756:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075489.828763:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075489.828787:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075489.828802:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923618304/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075489.828831:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075489.828834:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075489.828835:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075489.828839:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2179 00000100:00080000:2.0:1769075489.828842:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075489.828846:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075489.828854:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075489.828867:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923618432/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075489.828880:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075489.828898:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075489.828914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075489.828918:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2179 00000100:00080000:2.0:1769075489.828921:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075489.828924:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075489.828928:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075489.828939:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923618560/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075489.828952:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075489.828956:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075489.828957:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075489.828962:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2179 00000100:00080000:2.0:1769075489.828965:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075489.828968:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075489.828972:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075489.828983:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923618688/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075489.829003:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075489.829005:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075489.829007:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075489.829012:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2179 00000100:00080000:2.0:1769075489.829018:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075489.829024:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075489.829029:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075489.829048:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923618816/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075490.148658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075490.852732:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075491.172694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075491.876785:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075492.196711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075492.900741:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075493.220678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075493.924789:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075494.244720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075494.948714:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075494.948718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075494.948725:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075494.948729:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075494.948737:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2184 00000100:00080000:3.0:1769075494.948746:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075494.948751:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075494.948783:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923618944/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075494.948830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075494.948833:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075494.948850:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075494.948854:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2184 00000100:00080000:3.0:1769075494.948858:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075494.948861:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075494.948869:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075494.948880:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923619072/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075494.948894:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075494.948909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075494.948910:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075494.948914:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2184 00000100:00080000:3.0:1769075494.948916:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075494.948919:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075494.948922:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075494.948935:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923619200/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075494.948940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075494.948943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075494.948944:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075494.948947:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2184 00000100:00080000:3.0:1769075494.948950:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075494.948952:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075494.948956:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075494.948982:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923619328/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075494.949004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075494.949008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075494.949011:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075494.949016:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2184 00000100:00080000:3.0:1769075494.949022:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075494.949029:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075494.949039:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075494.949055:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923619456/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075495.268724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075495.972816:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075496.292709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075496.996743:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075497.316726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075498.020820:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075498.340722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075499.044761:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075499.364671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075500.068696:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075500.068722:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075500.068729:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075500.068743:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2189 00000100:00080000:3.0:1769075500.068757:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075500.068766:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075500.068814:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923619584/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075500.068820:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075500.068849:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075500.068854:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075500.068856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075500.068879:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2189 00000100:00080000:3.0:1769075500.068884:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075500.068889:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075500.068901:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075500.068920:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923619712/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075500.068933:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075500.068937:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075500.068939:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075500.068943:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2189 00000100:00080000:3.0:1769075500.068947:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075500.068952:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075500.068957:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075500.068969:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923619840/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075500.068980:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075500.068984:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075500.068986:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075500.068990:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2189 00000100:00080000:3.0:1769075500.068995:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075500.068999:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075500.069005:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075500.069018:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923619968/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075500.069037:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075500.069045:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075500.069049:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075500.069058:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2189 00000100:00080000:3.0:1769075500.069064:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075500.069069:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075500.069075:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075500.069093:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923620096/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075500.388651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075501.092802:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075501.412722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075502.116728:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075502.436656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075503.140827:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075503.141088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075500/real 1769075503] req@ffff969e0a7fd180 x1855007923619968/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075503.141116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923619968/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075503.141129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075503.141142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075503.141146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923619968:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075503.141155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075500/real 1769075503] req@ffff969e0a7fea00 x1855007923620096/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075503.141169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923620096/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075503.141177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075503.141180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075503.141189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620096:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075503.141200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075500/real 1769075503] req@ffff969e0a7fe680 x1855007923619712/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075503.141214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923619712/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075503.141223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075503.141227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075503.141231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923619712:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075503.141242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075500/real 1769075503] req@ffff969e0a7fc700 x1855007923619840/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075503.141255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923619840/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075503.141263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075503.141275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075503.141279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923619840:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075503.141286:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075500/real 1769075503] req@ffff969e0a7fed80 x1855007923619584/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075503.141300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923619584/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075503.141308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075503.141311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075503.141315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923619584:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075503.460708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075504.164848:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075504.484729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075505.188715:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075505.188724:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075505.188727:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075505.188737:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2194 00000100:00080000:3.0:1769075505.188746:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075505.188752:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075505.188781:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075505.188783:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923620224/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075505.188804:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075505.188824:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075505.188827:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075505.188832:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2194 00000100:00080000:3.0:1769075505.188836:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075505.188839:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075505.188845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923620224:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075505.188866:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075505.188907:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923620352/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075505.188924:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075505.188931:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075505.188934:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075505.188956:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2194 00000800:00000100:2.0:1769075505.188962:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075505.188964:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075505.188971:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075505.188977:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075505.188990:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923620480/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075505.189000:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075505.189004:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075505.189006:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075505.189013:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2194 00000100:00100000:2.0:1769075505.189019:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075505/real 1769075505] req@ffff969e0a7fd880 x1855007923620224/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075505.189022:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075505.189032:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075505.189041:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075505.189052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923620224/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075505.189057:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923620608/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075505.189107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075505.189109:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075505.189120:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075505.189147:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769075505.189149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923620224:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075505.189159:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075505.189169:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2194 00000100:00080000:3.0:1769075505.189175:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075505.189180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620352:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075505.189183:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075505.189189:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075505.189204:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923620736/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075505.189226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075505.189245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075505/real 1769075505] req@ffff969e0a7fed80 x1855007923620352/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075505.189259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923620352/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075505.189268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075505.189272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075505.189276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620352:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075505.189282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620480:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075505.189303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075505.189314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075505/real 1769075505] req@ffff969e0a7fc700 x1855007923620480/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075505.189326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923620480/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075505.189333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075505.189336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075505.189339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620480:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075505.189345:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620608:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075505.189364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075505.189374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075505/real 1769075505] req@ffff969e0a7fe680 x1855007923620608/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075505.189400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923620608/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075505.189408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075505.189411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075505.189415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620608:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075505.189441:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620736:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075505.189463:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075505.189475:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075505/real 1769075505] req@ffff969e0a7fea00 x1855007923620736/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075505.189488:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923620736/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075505.189497:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075505.189500:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075505.189504:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620736:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075505.508708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075506.212774:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075506.532750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075507.236829:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075507.556702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075508.260795:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075508.580677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075509.284737:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075509.604668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075510.308646:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075510.308655:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075510.308659:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075510.308667:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2199 00000100:00080000:3.0:1769075510.308677:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075510.308684:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075510.308713:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075510.308717:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923620864/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075510.308739:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075510.308741:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075510.308743:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075510.308747:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2199 00000100:00080000:3.0:1769075510.308750:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075510.308753:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075510.308761:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075510.308767:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923620864:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075510.308773:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923620992/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075510.308807:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075510.308809:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075510.308810:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075510.308813:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2199 00000100:00080000:3.0:1769075510.308817:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075510.308820:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075510.308824:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075510.308828:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075510.308839:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923621120/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075510.308852:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075510.308857:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075510/real 1769075510] req@ffff969e0a7fd180 x1855007923620864/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075510.308857:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075510.308861:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075510.308869:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2199 00000100:00080000:3.0:1769075510.308876:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075510.308880:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075510.308885:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075510.308886:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923620864/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075510.308897:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923621248/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075510.308898:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075510.308906:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075510.308914:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923620864:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075510.308916:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075510.308920:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075510.308922:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075510.308927:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2199 00000100:00100000:2.0:1769075510.308936:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620992:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075510.308937:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075510.308947:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075510.308954:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075510.308960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075510.308970:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075510/real 1769075510] req@ffff969e0a7fe300 x1855007923620992/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075510.308975:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923621376/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075510.308987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923620992/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075510.308998:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075510.309002:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075510.309004:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923620992:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075510.309010:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621120:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075510.309032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075510.309039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075510/real 1769075510] req@ffff969e0a7fc380 x1855007923621120/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075510.309054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923621120/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075510.309060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075510.309062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075510.309064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621120:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075510.309068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621248:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075510.309081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075510.309087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075510/real 1769075510] req@ffff969e0a7fdf80 x1855007923621248/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075510.309094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923621248/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075510.309099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075510.309101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075510.309103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621248:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075510.309113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621376:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075510.309123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075510.309129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075510/real 1769075510] req@ffff969e0a7fce00 x1855007923621376/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075510.309144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923621376/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075510.309158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075510.309161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075510.309163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621376:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075510.628710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075511.332787:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075511.652790:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075512.356719:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075512.676707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075513.380771:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075513.700790:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075514.404722:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075514.724703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075515.428679:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075515.428705:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075515.428709:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075515.428730:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2204 00000100:00080000:0.0:1769075515.428740:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075515.428745:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075515.428787:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075515.428790:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923621504/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075515.428812:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075515.428814:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075515.428816:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075515.428819:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2204 00000100:00080000:0.0:1769075515.428822:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075515.428825:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075515.428848:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075515.428855:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923621504:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075515.428860:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923621632/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075515.428870:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075515.428872:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075515.428873:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075515.428876:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2204 00000100:00080000:0.0:1769075515.428879:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075515.428882:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075515.428886:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075515.428894:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923621760/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075515.428910:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075515.428913:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075515.428915:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769075515.428917:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075515.428921:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2204 00000100:00080000:0.0:1769075515.428926:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075515.428930:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075515.428934:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075515.428947:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923621888/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075515.428950:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075515/real 1769075515] req@ffff969e06a60380 x1855007923621504/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075515.428977:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075515.428981:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923621504/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075515.428985:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075515.428990:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075515.428995:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075515.429000:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075515.429007:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2204 00000100:00100000:2.0:1769075515.429008:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923621504:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075515.429017:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075515.429023:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075515.429027:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075515.429032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621632:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075515.429040:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923622016/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075515.429052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075515.429073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075515/real 1769075515] req@ffff969e06a62d80 x1855007923621632/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075515.429082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923621632/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075515.429088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075515.429090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075515.429109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621632:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075515.429113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621760:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075515.429125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075515.429131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075515/real 1769075515] req@ffff969e06a61500 x1855007923621760/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075515.429139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923621760/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075515.429144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075515.429145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075515.429147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621760:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075515.429151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621888:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075515.429160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075515.429166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075515/real 1769075515] req@ffff969e06a61180 x1855007923621888/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075515.429173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923621888/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075515.429183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075515.429185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075515.429187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923621888:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075515.429198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923622016:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075515.429211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075515.429217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075515/real 1769075515] req@ffff969e06a62680 x1855007923622016/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075515.429224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923622016/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075515.429229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075515.429231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075515.429233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923622016:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075515.748747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075516.452717:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075516.772675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075517.476793:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075517.796677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075518.500750:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075518.820678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075519.524762:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075519.844701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075520.548715:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075520.548726:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075520.548730:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075520.548740:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2209 00000100:00080000:0.0:1769075520.548776:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075520.548794:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075520.548828:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923622144/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075520.548846:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075520.548860:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075520.548866:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075520.548868:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075520.548872:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2209 00000100:00080000:0.0:1769075520.548875:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075520.548880:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075520.548890:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075520.548901:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923622272/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075520.548916:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075520.548924:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075520.548926:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075520.548930:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2209 00000100:00080000:0.0:1769075520.548933:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075520.548936:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075520.548942:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075520.548954:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923622400/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075520.548963:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075520.548974:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075520.548976:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075520.548980:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2209 00000100:00080000:0.0:1769075520.548984:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075520.548993:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075520.549004:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075520.549019:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923622528/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075520.549029:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075520.549036:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075520.549039:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075520.549052:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2209 00000100:00080000:0.0:1769075520.549062:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075520.549066:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075520.549071:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075520.549097:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923622656/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075520.868702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075521.572863:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075521.892728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075522.596896:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075522.916684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075523.620799:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075523.940724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075524.644800:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075524.964766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075525.668719:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075525.668736:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075525.668744:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075525.668759:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2214 00000100:00080000:3.0:1769075525.668775:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769075525.668784:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075525.668784:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075525.668836:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923622784/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075525.668866:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075525.668869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075525.668871:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075525.668875:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2214 00000100:00080000:3.0:1769075525.668878:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075525.668882:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075525.668892:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075525.668902:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923622912/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075525.668963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075525.669013:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075525.669016:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075525.669023:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2214 00000100:00080000:3.0:1769075525.669061:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075525.669071:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075525.669082:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075525.669104:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923623040/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075525.669143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075525.669167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075525.669191:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075525.669206:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2214 00000100:00080000:3.0:1769075525.669219:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075525.669228:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075525.669236:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075525.669255:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923623168/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075525.669273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075525.669325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075525.669331:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075525.669338:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2214 00000100:00080000:3.0:1769075525.669370:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075525.669382:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075525.669396:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075525.669412:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923623296/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075525.988696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075526.692865:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075527.012679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075527.716708:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075528.036712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075528.740964:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075529.060751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075529.764784:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075530.084763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075530.788795:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769075530.788799:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075530.788809:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075530.788816:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075530.788828:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2219 00000100:00080000:0.0:1769075530.788842:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075530.788850:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075530.788915:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923623424/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075530.788969:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075530.788973:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075530.788976:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075530.788981:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2219 00000100:00080000:0.0:1769075530.788985:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075530.788990:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075530.789002:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075530.789015:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923623552/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075530.789026:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075530.789031:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075530.789034:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075530.789041:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2219 00000100:00080000:0.0:1769075530.789048:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075530.789053:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075530.789057:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075530.789068:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923623680/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075530.789077:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075530.789081:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075530.789082:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075530.789086:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2219 00000100:00080000:0.0:1769075530.789098:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075530.789102:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075530.789109:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075530.789119:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923623808/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075530.789129:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075530.789132:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075530.789134:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075530.789137:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2219 00000100:00080000:0.0:1769075530.789140:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075530.789143:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075530.789150:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075530.789165:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923623936/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075531.108682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075531.812844:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075532.132800:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075532.836987:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075533.156713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075533.860813:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075534.180732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075534.884738:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075535.204718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075535.908696:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075535.908707:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075535.908712:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075535.908725:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2225 00000100:00080000:0.0:1769075535.908737:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075535.908746:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075535.908786:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923624064/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075535.908811:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769075535.908819:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075535.908833:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075535.908836:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075535.908842:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2225 00000100:00080000:0.0:1769075535.908846:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075535.908856:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075535.908871:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075535.908892:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923624192/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075535.908908:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075535.908912:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075535.908914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075535.908918:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2225 00000100:00080000:0.0:1769075535.908922:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075535.908926:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075535.908931:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075535.908945:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923624320/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075535.908955:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075535.908960:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075535.908961:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075535.908965:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2225 00000100:00080000:0.0:1769075535.908970:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075535.908974:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075535.908986:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075535.909004:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923624448/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075535.909020:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075535.909024:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075535.909027:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075535.909032:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2225 00000100:00080000:0.0:1769075535.909039:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075535.909046:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075535.909053:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075535.909072:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923624576/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075536.228711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075536.932858:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075537.252809:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075537.956776:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075538.276664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075538.980788:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075539.300671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075540.004762:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075540.324691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075541.028835:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075541.028857:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075541.028866:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769075541.028873:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075541.028902:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2230 00000100:00080000:3.0:1769075541.028924:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075541.028937:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075541.029004:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923624704/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075541.029068:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075541.029072:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075541.029075:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075541.029081:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2230 00000100:00080000:3.0:1769075541.029086:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075541.029092:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075541.029107:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075541.029129:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923624832/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075541.029151:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075541.029165:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075541.029174:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075541.029190:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2230 00000100:00080000:3.0:1769075541.029221:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075541.029228:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075541.029236:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075541.029252:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923624960/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075541.029263:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075541.029267:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075541.029269:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075541.029273:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2230 00000100:00080000:3.0:1769075541.029276:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075541.029280:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075541.029286:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075541.029296:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923625088/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075541.029306:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075541.029321:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075541.029324:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075541.029340:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2230 00000100:00080000:3.0:1769075541.029348:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075541.029355:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075541.029362:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075541.029388:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923625216/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075541.348753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075542.052886:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075542.372684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075543.076827:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075543.396714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075544.100877:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075544.420701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075545.124889:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075545.444776:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075546.148690:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075546.148699:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075546.148704:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075546.148713:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2235 00000100:00080000:2.0:1769075546.148722:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075546.148729:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075546.148736:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075546.148784:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923625344/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075546.148844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075546.148847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075546.148848:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075546.148852:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2235 00000100:00080000:2.0:1769075546.148854:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075546.148874:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075546.148884:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075546.148919:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923625472/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075546.148957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075546.148962:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075546.148965:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075546.148986:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2235 00000100:00080000:2.0:1769075546.148991:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075546.148994:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075546.148999:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075546.149011:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923625600/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075546.149019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075546.149022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075546.149024:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075546.149027:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2235 00000100:00080000:2.0:1769075546.149029:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075546.149032:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075546.149040:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075546.149051:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923625728/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075546.149060:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075546.149068:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075546.149069:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075546.149076:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2235 00000100:00080000:2.0:1769075546.149084:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075546.149091:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075546.149097:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075546.149108:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923625856/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075546.468711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075547.172741:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075547.492740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075548.196812:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075548.516787:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075549.220883:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075549.540757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075550.244965:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075550.564701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075551.268743:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075551.268758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075551.268763:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075551.268794:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2240 00000100:00080000:0.0:1769075551.268812:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075551.268823:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075551.268871:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075551.268919:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923625984/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075551.268958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075551.268964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075551.268967:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075551.268974:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2240 00000100:00080000:0.0:1769075551.269000:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075551.269007:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075551.269019:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075551.269038:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923626112/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075551.269053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075551.269057:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075551.269059:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075551.269064:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2240 00000100:00080000:0.0:1769075551.269069:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075551.269073:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075551.269080:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075551.269092:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923626240/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075551.269127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075551.269132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075551.269135:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075551.269140:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2240 00000100:00080000:0.0:1769075551.269145:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075551.269151:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075551.269169:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075551.269181:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923626368/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075551.269210:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075551.269215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075551.269218:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075551.269223:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2240 00000100:00080000:0.0:1769075551.269228:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075551.269233:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075551.269239:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075551.269254:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923626496/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075551.588722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075552.292910:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075552.612669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075553.316717:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075554.340736:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075555.364743:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075556.388755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075556.388786:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075556.388792:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075556.388803:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2245 00000100:00080000:0.0:1769075556.388815:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075556.388835:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075556.388873:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923626624/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075556.388904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769075556.388907:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075556.388910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075556.388913:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075556.388921:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2245 00000100:00080000:0.0:1769075556.388927:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075556.388934:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075556.388976:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075556.389001:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923626752/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075556.389021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075556.389024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075556.389027:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075556.389039:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2245 00000100:00080000:0.0:1769075556.389047:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075556.389053:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075556.389059:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075556.389076:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923626880/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075556.389089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075556.389099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075556.389102:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075556.389112:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2245 00000100:00080000:0.0:1769075556.389119:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075556.389124:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075556.389129:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075556.389142:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923627008/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075556.389170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075556.389178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075556.389191:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075556.389202:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2245 00000100:00080000:0.0:1769075556.389209:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075556.389220:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075556.389226:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075556.389238:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923627136/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075557.412905:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075558.436860:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075559.460904:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075560.484871:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075561.508702:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075561.508711:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075561.508716:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075561.508724:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2250 00000100:00080000:0.0:1769075561.508734:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075561.508741:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075561.508775:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923627264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075561.508808:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075561.508811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075561.508813:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075561.508817:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2250 00000100:00080000:0.0:1769075561.508821:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075561.508824:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075561.508827:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:0.0:1769075561.508832:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075561.508842:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923627392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075561.508866:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075561.508869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075561.508870:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075561.508874:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2250 00000100:00100000:2.0:1769075561.508879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923627264:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075561.508880:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075561.508887:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075561.508891:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075561.508900:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923627520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075561.508907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075561.508910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075561.508911:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075561.508914:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2250 00000100:00080000:0.0:1769075561.508917:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075561.508920:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075561.508923:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075561.508932:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923627648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075561.508940:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075561.508947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075561.508951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075561.508954:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075561.508959:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2250 00000100:00080000:0.0:1769075561.508971:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075561.508972:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075561/real 1769075561] req@ffff969e06a62d80 x1855007923627264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075561.508980:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075561.508991:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075561.509002:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923627264/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075561.509006:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923627776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075561.509035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075561.509047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075561.509051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923627264:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075561.509059:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923627392:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075561.509081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075561.509090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075561/real 1769075561] req@ffff969e06a60380 x1855007923627392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075561.509098:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923627392/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075561.509104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075561.509106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075561.509108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923627392:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075561.509129:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923627520:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075561.509149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075561.509160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075561/real 1769075561] req@ffff969e06a61f80 x1855007923627520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075561.509173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923627520/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075561.509180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075561.509182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075561.509185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923627520:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075561.509189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923627648:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075561.509210:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075561.509218:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075561/real 1769075561] req@ffff969e06a60a80 x1855007923627648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075561.509225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923627648/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075561.509230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075561.509232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075561.509234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923627648:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075561.509238:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923627776:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075561.509247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075561.509253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075561/real 1769075561] req@ffff969e06a63800 x1855007923627776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075561.509259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923627776/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075561.509264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075561.509266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075561.509268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923627776:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075562.532832:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075563.556866:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075564.580832:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075565.604856:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075566.628977:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075566.629146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923627904:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075566.629314:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628032:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075566.629429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628160:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075566.629523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628288:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075566.629803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628416:192.168.204.143@tcp:8:kworker.0 00010000:00010000:3.0:1769075567.652830:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075568.676900:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769075569.700770:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769075569.700860:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769075569.700892:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769075569.700908:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075569.700955:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075569.700975:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075569.700983:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075569.700992:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075569.701060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075566/real 1769075569] req@ffff969e10b61500 x1855007923628416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075569.701086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923628416/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075569.701099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075569.701105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075569.701110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628416:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075569.701120:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075566/real 1769075569] req@ffff969e10b61f80 x1855007923628288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075569.701131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923628288/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075569.701139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075569.701143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075569.701146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628288:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075569.701153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075566/real 1769075569] req@ffff969e10b60a80 x1855007923628160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075569.701190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923628160/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075569.701196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075569.701198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075569.701200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628160:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075569.701205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075566/real 1769075569] req@ffff969e10b62a00 x1855007923628032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075569.701214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923628032/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075569.701222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075569.701226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075569.701229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628032:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075569.701236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075566/real 1769075569] req@ffff969e10b62680 x1855007923627904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075569.701249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923627904/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075569.701256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075569.701260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075569.701263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923627904:192.168.204.143@tcp:250:kworker.0 00010000:00010000:2.0:1769075570.724853:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075571.748779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075571.748791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1769075571.748792:0:32724:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075571.748796:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075571.748808:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2260 00000100:00080000:3.0:1769075571.748820:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075571.748828:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075571.748880:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923628544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075571.748906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075571.748909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075571.748910:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075571.748943:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2260 00000100:00100000:2.0:1769075571.748945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923628544:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075571.748949:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075571.748955:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075571.748968:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075571.748988:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923628672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075571.749001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075571.749004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075571.749006:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075571.749011:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2260 00000100:00080000:3.0:1769075571.749018:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075571.749023:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075571.749029:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075571.749034:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075571.749047:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923628800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075571.749061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075571.749066:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075571.749068:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075571.749074:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2260 00000100:00080000:3.0:1769075571.749079:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075571.749084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075571/real 1769075571] req@ffff969e0a7fca80 x1855007923628544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075571.749087:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075571.749098:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075571.749116:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923628928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075571.749119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923628544/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075571.749141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075571.749145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075571.749151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075571.749152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075571.749159:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075571.749163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923628544:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075571.749167:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2260 00000100:00080000:3.0:1769075571.749177:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075571.749181:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075571.749188:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075571.749203:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923629056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075571.749223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628672:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075571.749255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075571.749271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075571/real 1769075571] req@ffff969e0a7fce00 x1855007923628672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075571.749284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923628672/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075571.749294:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075571.749298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075571.749302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628672:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075571.749309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628800:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075571.749333:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075571.749345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075571/real 1769075571] req@ffff969e0a7fdf80 x1855007923628800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075571.749368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923628800/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075571.749378:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075571.749382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075571.749386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628800:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075571.749393:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628928:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075571.749418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075571.749431:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075571/real 1769075571] req@ffff969e0a7fc380 x1855007923628928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075571.749445:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923628928/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075571.749455:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075571.749460:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075571.749464:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923628928:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075571.749470:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923629056:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075571.749493:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075571.749518:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075571/real 1769075571] req@ffff969e0a7fe300 x1855007923629056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075571.749534:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923629056/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075571.749575:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075571.749580:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075571.749584:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923629056:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769075572.772849:0:32714:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075573.797060:0:32724:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 99 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1769075573.797080:0:32724:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e06aae400/0x44e2946041b61945 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b91fc930f88de39 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1769075573.797136:0:32724:0:(osc_cache.c:1892:try_to_add_extent_for_io()) ### extent: 000000008f49cd8b ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e06aae400/0x44e2946041b61945 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88de39 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1769075573.797159:0:32724:0:(osc_cache.c:1103:osc_extent_make_ready()) ### extent: 000000008f49cd8b ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e06aae400/0x44e2946041b61945 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88de39 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1769075573.797233:0:32714:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 99 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1769075573.797253:0:32724:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e06aae400/0x44e2946041b61945 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88de39 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1769075573.797387:0:32724:0:(osc_request.c:1967:osc_brw_prep_request()) brw rpc ffff969e0a46c700 - object 0x280000401:559 offset 0<>8 00000100:00080000:3.0:1769075573.797558:0:2420:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff969e0a46c700 x1855007923629184/t0(0) o4->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:6/4 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1769075573.797796:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075573.797801:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075573.797805:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769075573.797837:0:32714:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e06aaec00/0x44e2946041b6195a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b91fc930f88de71 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075573.797842:0:32795:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 220 locks of 0 total in blp. (0 blwis in pool) 00000008:00010000:2.0:1769075573.797874:0:32714:0:(osc_cache.c:1892:try_to_add_extent_for_io()) ### extent: 00000000fd782c19 ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e06aaec00/0x44e2946041b6195a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88de71 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1769075573.797887:0:32714:0:(osc_cache.c:1103:osc_extent_make_ready()) ### extent: 00000000fd782c19 ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e06aaec00/0x44e2946041b6195a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88de71 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1769075573.797952:0:32714:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e06aaec00/0x44e2946041b6195a lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88de71 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1769075573.798040:0:32714:0:(osc_request.c:1967:osc_brw_prep_request()) brw rpc ffff969e0a46c000 - object 0x2c0000401:559 offset 0<>8 00010000:00010000:0.0:1769075573.798122:0:32796:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075573.798124:0:2419:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff969e0a46c000 x1855007923629312/t0(0) o4->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:6/4 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1769075573.798128:0:32796:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075573.798133:0:32796:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075573.807836:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05775200/0x44e2946041b60a11 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b5b3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807857:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05655e00/0x44e2946041b609fc lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b57b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807865:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00030c00/0x44e2946041b609e7 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b543 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807873:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066afc00/0x44e2946041b609d2 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b50b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807880:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066afa00/0x44e2946041b609bd lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b4d3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807888:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066aec00/0x44e2946041b609a8 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b49b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807897:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066ae600/0x44e2946041b60993 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b463 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807905:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066aee00/0x44e2946041b6097e lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b42b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807914:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066af200/0x44e2946041b60969 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b3f3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807923:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066ae800/0x44e2946041b60954 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b3bb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807932:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05655400/0x44e2946041b6093f lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b383 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807954:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05654600/0x44e2946041b6092a lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b34b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807963:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05654000/0x44e2946041b60915 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b313 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807971:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05654400/0x44e2946041b60900 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b2db expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807980:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05655a00/0x44e2946041b608eb lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b2a3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.807997:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05655c00/0x44e2946041b608d6 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b26b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808006:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05654200/0x44e2946041b608c1 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b233 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808015:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00031800/0x44e2946041b608ac lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b1fb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808025:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00030e00/0x44e2946041b60897 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b1c3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808033:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00031600/0x44e2946041b60882 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b18b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808041:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00031c00/0x44e2946041b6086d lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b153 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808064:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e01668c00/0x44e2946041b60858 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b11b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808082:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e01668a00/0x44e2946041b60843 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b0e3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808091:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0400/0x44e2946041b6082e lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b0ab expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808100:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0c00/0x44e2946041b60819 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b073 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808108:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0600/0x44e2946041b60804 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b03b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808116:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0800/0x44e2946041b607ef lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b003 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808125:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e1e00/0x44e2946041b607da lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88afcb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808134:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0200/0x44e2946041b607c5 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88af93 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808144:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d0200/0x44e2946041b607b0 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88af5b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808153:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d1000/0x44e2946041b6079b lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88af23 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808172:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d1e00/0x44e2946041b60786 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88aeeb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808181:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d1400/0x44e2946041b60771 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88aeb3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808190:0:32795:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d0600/0x44e2946041b6075c lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ae7b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808199:0:32795:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 220 locks packed 00010000:00010000:1.0:1769075573.808236:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aafc00/0x44e2946041b61953 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88de63 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808265:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aaea00/0x44e2946041b6193e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88de2b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00000100:00080000:0.0:1769075573.808267:0:2417:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff969e10b63800 x1855007923629440/t0(0) o103->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:17/18 lens 2072/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1769075573.808310:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aaf600/0x44e2946041b61929 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ddf3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808328:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a2da00/0x44e2946041b61914 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ddbb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808347:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a2cc00/0x44e2946041b618ff lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dd83 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808373:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a2c400/0x44e2946041b618ea lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dd4b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808391:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a2c200/0x44e2946041b618d5 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dd13 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808415:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a2d600/0x44e2946041b618c0 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dcdb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808430:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a2d400/0x44e2946041b618ab lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dca3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808444:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a2de00/0x44e2946041b61896 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dc6b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808458:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a2c000/0x44e2946041b61881 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dc33 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808493:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aa0200/0x44e2946041b6186c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dbfb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808508:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057fca00/0x44e2946041b61857 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dbc3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808522:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057fc200/0x44e2946041b61842 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88db8b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808537:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057fd600/0x44e2946041b6182d lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88db53 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808551:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057fc600/0x44e2946041b61818 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88db1b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808566:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057fc800/0x44e2946041b61803 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88dae3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808610:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057fc000/0x44e2946041b617ee lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88daab expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808627:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057fcc00/0x44e2946041b617d9 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88da73 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808640:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06acbc00/0x44e2946041b617c4 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88da3b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808655:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aa0400/0x44e2946041b617af lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88da03 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808669:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aa0c00/0x44e2946041b6179a lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d9cb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808684:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aa1400/0x44e2946041b61785 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d993 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808716:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aa0a00/0x44e2946041b61770 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d95b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808732:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aa1600/0x44e2946041b6175b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d923 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808761:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aa1200/0x44e2946041b61746 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d8eb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808779:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aa0800/0x44e2946041b61731 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d8b3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808809:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06acae00/0x44e2946041b6171c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d87b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808827:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a7dc00/0x44e2946041b61707 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d843 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808852:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aca200/0x44e2946041b616f2 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d80b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808869:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06acb600/0x44e2946041b616dd lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d7d3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808886:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aca600/0x44e2946041b616c8 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d79b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808901:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06acbe00/0x44e2946041b616b3 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d763 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808917:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06aca000/0x44e2946041b6169e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d72b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808932:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06acac00/0x44e2946041b61689 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d6f3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808948:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06acaa00/0x44e2946041b61674 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d6bb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.808965:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0572d400/0x44e2946041b6165f lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d683 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809000:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0572de00/0x44e2946041b6164a lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d64b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809017:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0572d000/0x44e2946041b61635 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d613 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809032:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0572da00/0x44e2946041b61620 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d5db expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809062:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0572cc00/0x44e2946041b6160b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d5a3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809087:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0572c400/0x44e2946041b615f6 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d56b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809100:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0572ce00/0x44e2946041b615e1 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d533 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809113:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06ac2600/0x44e2946041b615cc lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d4fb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809128:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06ac2200/0x44e2946041b615b7 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d4c3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809140:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06ac3200/0x44e2946041b615a2 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d48b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809154:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06ac2000/0x44e2946041b6158d lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d453 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809177:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06ac3000/0x44e2946041b61578 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d41b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809191:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06ac3a00/0x44e2946041b61563 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d3e3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809205:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06ac2c00/0x44e2946041b6154e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d3ab expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809219:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06ac2a00/0x44e2946041b61539 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d373 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809233:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a8ae00/0x44e2946041b61524 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d33b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809247:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a83c00/0x44e2946041b6150f lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d303 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809262:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a7ca00/0x44e2946041b614fa lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d2cb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809275:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a7ce00/0x44e2946041b614e5 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d293 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809299:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a7d400/0x44e2946041b614d0 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xcf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d25b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809313:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a7d200/0x44e2946041b614bb lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d223 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809336:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a7c800/0x44e2946041b614a6 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d1eb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809351:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a7da00/0x44e2946041b61491 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d1b3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809375:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a83400/0x44e2946041b6147c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d17b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809388:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a8bc00/0x44e2946041b61467 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d143 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809402:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a8a400/0x44e2946041b61452 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d10b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809414:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a8a200/0x44e2946041b6143d lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d0d3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809428:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a8b400/0x44e2946041b61428 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d09b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809441:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a8b200/0x44e2946041b61413 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d063 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809455:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a8a800/0x44e2946041b613fe lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88d02b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809469:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a8ba00/0x44e2946041b613e9 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cff3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809509:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067cc200/0x44e2946041b613d4 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cfbb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809523:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a83a00/0x44e2946041b613bf lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cf83 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809537:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a82c00/0x44e2946041b613aa lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cf4b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809552:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a82400/0x44e2946041b61395 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cf13 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809565:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a82200/0x44e2946041b61380 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cedb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809578:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a83600/0x44e2946041b6136b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cea3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809613:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a83200/0x44e2946041b61356 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ce6b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809629:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a82800/0x44e2946041b61341 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ce33 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809641:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a37400/0x44e2946041b6132c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cdfb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809655:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067cca00/0x44e2946041b61317 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cdc3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809680:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067cce00/0x44e2946041b61302 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cd8b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809693:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067cda00/0x44e2946041b612ed lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cd53 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809707:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067cc800/0x44e2946041b612d8 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cd1b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809721:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067cdc00/0x44e2946041b612c3 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cce3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809734:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067cc000/0x44e2946041b612ae lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ccab expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809759:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067ccc00/0x44e2946041b61299 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cc73 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809773:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a36e00/0x44e2946041b61284 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cc3b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809786:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a36a00/0x44e2946041b6126f lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cc03 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809799:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a37a00/0x44e2946041b6125a lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cbcb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809814:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a37000/0x44e2946041b61245 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xb0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cb93 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809848:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a36600/0x44e2946041b61230 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xaf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cb5b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809862:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a36800/0x44e2946041b6121b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cb23 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809876:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a36000/0x44e2946041b61206 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xad:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88caeb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809889:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a36c00/0x44e2946041b611f1 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88cab3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809903:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057cc000/0x44e2946041b611dc lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xab:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ca7b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809916:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05723600/0x44e2946041b611c7 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ca43 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809931:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05723400/0x44e2946041b611b2 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ca0b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809944:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05723e00/0x44e2946041b6119d lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c9d3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.809994:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05723000/0x44e2946041b61188 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c99b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810012:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05723a00/0x44e2946041b61173 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c963 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810035:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05722c00/0x44e2946041b6115e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c92b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810049:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05722a00/0x44e2946041b61149 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c8f3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810063:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0564cc00/0x44e2946041b61134 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c8bb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810089:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1a600/0x44e2946041b6111f lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c883 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810102:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0564dc00/0x44e2946041b6110a lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c84b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810118:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0564ca00/0x44e2946041b610f5 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c813 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810132:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0564ce00/0x44e2946041b610e0 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c7db expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810147:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0564d400/0x44e2946041b610cb lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c7a3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810161:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0564c600/0x44e2946041b610b6 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x9d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c76b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810176:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0564d000/0x44e2946041b610a1 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c733 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810200:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e0564d800/0x44e2946041b6108c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c6fb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810214:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057cc800/0x44e2946041b61077 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c6c3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810228:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057cc600/0x44e2946041b61062 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c68b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810243:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057ccc00/0x44e2946041b6104d lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c653 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810258:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057cd000/0x44e2946041b61038 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c61b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810272:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057cd800/0x44e2946041b61023 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c5e3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810298:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057cdc00/0x44e2946041b6100e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c5ab expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810311:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057cc200/0x44e2946041b60ff9 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c573 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810325:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1b600/0x44e2946041b60fe4 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c53b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810340:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1ac00/0x44e2946041b60fcf lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c503 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810372:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1a400/0x44e2946041b60fba lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x91:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c4cb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810387:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1a200/0x44e2946041b60fa5 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c493 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810401:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1a800/0x44e2946041b60f90 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x8f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c45b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810416:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1b200/0x44e2946041b60f7b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c423 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810430:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1be00/0x44e2946041b60f66 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x8d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c3eb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810444:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e06a1a000/0x44e2946041b60f51 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x8c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c3b3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810458:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b9a800/0x44e2946041b60f3c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c37b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810490:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057a3000/0x44e2946041b60f27 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c343 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810507:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057a2800/0x44e2946041b60f12 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c30b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810522:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057a2200/0x44e2946041b60efd lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c2d3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810545:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057a3a00/0x44e2946041b60ee8 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c29b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810559:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057a3400/0x44e2946041b60ed3 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c263 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810574:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057a2a00/0x44e2946041b60ebe lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c22b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810588:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057a3600/0x44e2946041b60ea9 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c1f3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810632:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057d3000/0x44e2946041b60e94 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c1bb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810651:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b9b800/0x44e2946041b60e7f lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c183 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810692:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b9a000/0x44e2946041b60e6a lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c14b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810713:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b9ae00/0x44e2946041b60e55 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c113 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810728:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b9ac00/0x44e2946041b60e40 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c0db expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810742:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b9bc00/0x44e2946041b60e2b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c0a3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810777:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b9a200/0x44e2946041b60e16 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x7d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c06b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810791:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b9b400/0x44e2946041b60e01 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88c033 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810805:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05620e00/0x44e2946041b60dec lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bffb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810821:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057d2000/0x44e2946041b60dd7 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bfc3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810847:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057d3800/0x44e2946041b60dc2 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bf8b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810861:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057d3c00/0x44e2946041b60dad lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bf53 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810876:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057d2a00/0x44e2946041b60d98 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bf1b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810890:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057d2e00/0x44e2946041b60d83 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bee3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810903:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057d3400/0x44e2946041b60d6e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88beab expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810918:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057d3e00/0x44e2946041b60d59 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88be73 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810957:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05620400/0x44e2946041b60d44 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88be3b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810973:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057ea000/0x44e2946041b60d2f lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88be03 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.810999:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05620c00/0x44e2946041b60d1a lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bdcb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811014:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05621000/0x44e2946041b60d05 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x70:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bd93 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811028:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05620200/0x44e2946041b60cf0 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x6f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bd5b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811043:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05621600/0x44e2946041b60cdb lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bd23 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811070:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05621200/0x44e2946041b60cc6 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x6d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bceb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811085:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05620800/0x44e2946041b60cb1 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x6c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bcb3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811099:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b02c00/0x44e2946041b60c9c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x6b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bc7b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811114:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057ea400/0x44e2946041b60c87 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x6a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bc43 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811135:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057ea200/0x44e2946041b60c72 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bc0b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811149:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057eb600/0x44e2946041b60c5d lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x68:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bbd3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811164:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057eb200/0x44e2946041b60c48 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x67:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bb9b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811179:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057ebe00/0x44e2946041b60c33 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bb63 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811193:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057eb000/0x44e2946041b60c1e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88bb2b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811208:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e057eb800/0x44e2946041b60c09 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88baf3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811224:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05775000/0x44e2946041b60bf4 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88babb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811242:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b03c00/0x44e2946041b60bdf lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ba83 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811258:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b02a00/0x44e2946041b60bca lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ba4b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811274:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b02e00/0x44e2946041b60bb5 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ba13 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811309:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b03400/0x44e2946041b60ba0 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b9db expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811323:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b02600/0x44e2946041b60b8b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b9a3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811339:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b02800/0x44e2946041b60b76 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b96b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811354:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e10b03a00/0x44e2946041b60b61 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b933 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811379:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05775600/0x44e2946041b60b4c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b8fb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811395:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05700200/0x44e2946041b60b37 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b8c3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811410:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05701600/0x44e2946041b60b22 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b88b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811424:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05701200/0x44e2946041b60b0d lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b853 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811439:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05700400/0x44e2946041b60af8 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b81b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811454:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05701000/0x44e2946041b60ae3 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b7e3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811495:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05701a00/0x44e2946041b60ace lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b7ab expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811511:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05701c00/0x44e2946041b60ab9 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x54:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b773 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811526:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066af600/0x44e2946041b60aa4 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b73b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811544:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05774800/0x44e2946041b60a8f lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b703 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811560:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05775c00/0x44e2946041b60a7a lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b6cb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811578:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05775a00/0x44e2946041b60a65 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b693 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811619:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05774c00/0x44e2946041b60a50 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b65b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811637:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05774400/0x44e2946041b60a3b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b623 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811653:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05774e00/0x44e2946041b60a26 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b5eb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811669:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05775200/0x44e2946041b60a11 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b5b3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811693:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05655e00/0x44e2946041b609fc lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b57b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811710:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00030c00/0x44e2946041b609e7 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b543 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811728:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066afc00/0x44e2946041b609d2 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b50b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811743:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066afa00/0x44e2946041b609bd lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b4d3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811764:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066aec00/0x44e2946041b609a8 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b49b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811779:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066ae600/0x44e2946041b60993 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b463 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811793:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066aee00/0x44e2946041b6097e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b42b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811808:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066af200/0x44e2946041b60969 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b3f3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811824:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e066ae800/0x44e2946041b60954 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b3bb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811851:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05655400/0x44e2946041b6093f lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b383 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811877:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05654600/0x44e2946041b6092a lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b34b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811910:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05654000/0x44e2946041b60915 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b313 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811927:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05654400/0x44e2946041b60900 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b2db expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811942:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05655a00/0x44e2946041b608eb lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b2a3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811957:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05655c00/0x44e2946041b608d6 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b26b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811971:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e05654200/0x44e2946041b608c1 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b233 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.811997:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00031800/0x44e2946041b608ac lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b1fb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812014:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00030e00/0x44e2946041b60897 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b1c3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812028:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00031600/0x44e2946041b60882 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b18b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812042:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e00031c00/0x44e2946041b6086d lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b153 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812082:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e01668c00/0x44e2946041b60858 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b11b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812099:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e01668a00/0x44e2946041b60843 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b0e3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812113:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0400/0x44e2946041b6082e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b0ab expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812128:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0c00/0x44e2946041b60819 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b073 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812142:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0600/0x44e2946041b60804 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b03b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812154:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0800/0x44e2946041b607ef lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88b003 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812168:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e1e00/0x44e2946041b607da lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88afcb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812183:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e097e0200/0x44e2946041b607c5 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88af93 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812196:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d0200/0x44e2946041b607b0 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88af5b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812211:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d1000/0x44e2946041b6079b lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88af23 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812233:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d1e00/0x44e2946041b60786 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88aeeb expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812247:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d1400/0x44e2946041b60771 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88aeb3 expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812261:0:32795:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff969e12057000 lock: ffff969e067d0600/0x44e2946041b6075c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b91fc930f88ae7b expref: -99 pid: 31824 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1769075573.812292:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075573.812295:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075573.812298:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075574.116755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075574.820810:0:32796:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075575.140698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075575.844747:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075576.164694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075576.868681:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075576.868694:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075576.868697:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769075576.868700:0:32796:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075576.868706:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2266 00000100:00080000:3.0:1769075576.868715:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075576.868721:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075576.868761:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923629568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075576.868782:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075576.868794:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075576.868795:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075576.868799:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2266 00000100:00080000:3.0:1769075576.868812:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075576.868816:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075576.868825:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075576.868840:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923629696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075576.868841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923629568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075576.868856:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075576.868859:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075576.868860:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075576.868864:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2266 00000100:00080000:3.0:1769075576.868867:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075576.868869:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075576.868872:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075576.868882:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923629824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075576.868888:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075576.868891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075576.868892:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075576.868895:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2266 00000100:00080000:3.0:1769075576.868897:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075576.868899:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075576.868906:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075576.868921:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075576.868932:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923629952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075576.868940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075576.868943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075576.868945:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075576.868949:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2266 00000100:00100000:2.0:1769075576.868950:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075576/real 1769075576] req@ffff969e0a7fd180 x1855007923629568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075576.868952:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075576.868957:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075576.868960:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075576.868967:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923630080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075576.868997:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923629568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075576.869007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075576.869011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075576.869014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923629568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075576.869032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923629696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075576.869058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075576.869066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075576/real 1769075576] req@ffff969e0a7fea00 x1855007923629696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075576.869074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923629696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075576.869080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075576.869082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075576.869091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923629696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075576.869095:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923629824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075576.869109:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075576.869115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075576/real 1769075576] req@ffff969e0a7fe680 x1855007923629824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075576.869123:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923629824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075576.869128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075576.869130:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075576.869132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923629824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075576.869135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923629952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075576.869145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075576.869150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075576/real 1769075576] req@ffff969e0a7fc700 x1855007923629952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075576.869158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923629952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075576.869163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075576.869165:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075576.869166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923629952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075576.869170:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075576.869189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075576.869201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075576/real 1769075576] req@ffff969e0a7fed80 x1855007923630080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075576.869212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923630080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075576.869229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075576.869233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075576.869237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075577.188667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075577.892696:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075578.212710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075578.916791:0:32796:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075579.236752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075579.940712:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075580.260763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075580.964939:0:32796:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075581.284710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075581.988690:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075581.988701:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075581.988705:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075581.988716:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2271 00010000:00010000:1.0:1769075581.988721:0:32795:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075581.988727:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075581.988737:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075581.988776:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923630208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075581.988804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075581.988808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075581.988810:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075581.988814:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2271 00000100:00080000:3.0:1769075581.988834:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075581.988839:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075581.988850:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075581.988869:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923630336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075581.988872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923630208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075581.988892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075581.988896:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075581.988898:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075581.988903:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2271 00000100:00080000:3.0:1769075581.988906:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075581.988909:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075581.988914:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075581.988932:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923630464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075581.988946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075581.988953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075581.988956:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075581.988962:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075581.988970:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2271 00000100:00080000:3.0:1769075581.988978:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075581.988984:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075581.988993:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075581.988994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075581/real 1769075581] req@ffff969e0a7fd880 x1855007923630208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075581.989017:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923630592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075581.989021:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923630208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075581.989048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075581.989057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075581.989068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923630208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075581.989074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075581.989080:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075581.989082:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075581.989088:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2271 00000100:00080000:3.0:1769075581.989094:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075581.989100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630336:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075581.989102:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075581.989116:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075581.989132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075581.989140:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923630720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075581.989148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075581/real 1769075581] req@ffff969e0a7fed80 x1855007923630336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075581.989182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923630336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075581.989206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075581.989210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075581.989213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075581.989219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075581.989238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075581.989247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075581/real 1769075581] req@ffff969e0a7fc700 x1855007923630464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075581.989258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923630464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075581.989265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075581.989267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075581.989270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075581.989275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075581.989288:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075581.989296:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075581/real 1769075581] req@ffff969e0a7fe680 x1855007923630592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075581.989306:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923630592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075581.989313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075581.989315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075581.989325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075581.989364:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075581.989385:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075581.989404:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075581/real 1769075581] req@ffff969e0a7fea00 x1855007923630720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075581.989414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923630720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075581.989422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075581.989425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075581.989428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075582.308729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075583.012795:0:32796:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075583.332660:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075584.036735:0:31815:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1769075584.036760:0:31815:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1769075584.036779:0:32795:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1769075584.036803:0:32795:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e067d0000/0x44e2946041b60763 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b91fc930f88ae89 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1769075584.036806:0:32796:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1769075584.037235:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075584.037238:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769075584.037239:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769075584.037280:0:32796:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e067d0800/0x44e2946041b60778 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b91fc930f88aec1 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000100:00100000:1.0:1769075584.356692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075585.060712:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075585.380680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075586.084694:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075586.404688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075587.108656:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075587.108665:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075587.108669:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075587.108677:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2276 00000100:00080000:3.0:1769075587.108687:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769075587.108696:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075587.108696:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075587.108748:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923630848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075587.108777:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075587.108780:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075587.108782:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075587.108785:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2276 00000100:00080000:3.0:1769075587.108788:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075587.108791:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075587.108813:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075587.108823:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923630976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075587.108832:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923630848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075587.108835:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075587.108841:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075587.108842:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075587.108846:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2276 00000100:00080000:3.0:1769075587.108848:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075587.108851:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075587.108855:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075587.108862:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923631104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075587.108869:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075587.108872:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075587.108873:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075587.108875:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2276 00000100:00080000:3.0:1769075587.108884:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075587.108887:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075587.108891:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075587.108892:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075587.108903:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923631232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075587.108911:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075587.108913:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075587.108919:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075587.108923:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075587/real 1769075587] req@ffff969e0a7fd180 x1855007923630848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075587.108924:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2276 00000100:00080000:3.0:1769075587.108930:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075587.108937:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075587.108943:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075587.108955:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923631360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075587.108957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923630848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075587.108978:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075587.108984:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075587.108989:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923630848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075587.109013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075587.109036:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075587.109054:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075587/real 1769075587] req@ffff969e0a7fe300 x1855007923630976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075587.109063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923630976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075587.109070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075587.109072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075587.109075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923630976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075587.109079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075587.109093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075587.109099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075587/real 1769075587] req@ffff969e0a7fc380 x1855007923631104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075587.109107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923631104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075587.109112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075587.109114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075587.109116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075587.109120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075587.109131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075587.109137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075587/real 1769075587] req@ffff969e0a7fdf80 x1855007923631232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075587.109144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923631232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075587.109155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075587.109158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075587.109160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075587.109164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075587.109177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075587.109184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075587/real 1769075587] req@ffff969e0a7fce00 x1855007923631360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075587.109191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923631360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075587.109196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075587.109198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075587.109200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075587.428652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075588.132779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075588.452659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075589.156907:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075589.476683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075590.180699:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075590.500705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075591.204797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075591.524792:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075592.228697:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075592.228710:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1769075592.228717:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075592.228717:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075592.228732:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2281 00000100:00080000:3.0:1769075592.228745:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075592.228752:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075592.228799:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923631488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075592.228840:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075592.228845:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075592.228847:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075592.228852:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2281 00000100:00080000:3.0:1769075592.228856:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075592.228859:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075592.228870:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075592.228881:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923631616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075592.228913:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075592.228917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923631488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075592.228917:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075592.228921:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075592.228928:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2281 00000100:00080000:3.0:1769075592.228931:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075592.228934:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075592.228938:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075592.228949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923631744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075592.228956:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075592.228959:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075592.228961:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075592.228963:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2281 00000100:00080000:3.0:1769075592.228965:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075592.228968:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075592.228971:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075592.228981:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923631872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075592.228991:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075592.228995:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075592.228997:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075592.229002:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2281 00000800:00000100:2.0:1769075592.229005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075592.229018:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075592.229025:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075592.229031:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075592.229047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075592/real 1769075592] req@ffff969e0a7fca80 x1855007923631488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075592.229048:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923632000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075592.229086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923631488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075592.229096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075592.229100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075592.229104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923631488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075592.229137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075592.229155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075592.229182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075592/real 1769075592] req@ffff969e0a7fc000 x1855007923631616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075592.229192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923631616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075592.229198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075592.229201:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075592.229203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075592.229207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075592.229222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075592.229228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075592/real 1769075592] req@ffff969e0a7fd500 x1855007923631744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075592.229236:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923631744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075592.229241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075592.229243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075592.229245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075592.229248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075592.229260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075592.229266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075592/real 1769075592] req@ffff969e0a7fdc00 x1855007923631872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075592.229280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923631872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075592.229286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075592.229288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075592.229290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923631872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075592.229294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075592.229318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075592.229323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075592/real 1769075592] req@ffff969e0a7ff800 x1855007923632000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075592.229331:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923632000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075592.229336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075592.229338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075592.229340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075592.548731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075593.252899:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075593.572679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075594.276847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075594.596705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075595.300738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075595.620662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075596.324772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075596.644680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075597.348754:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075597.348766:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075597.348772:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075597.348784:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2286 00000100:00080000:2.0:1769075597.348797:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769075597.348804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075597.348805:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075597.348855:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923632128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075597.348908:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075597.348911:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075597.348913:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075597.348916:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2286 00000100:00080000:2.0:1769075597.348919:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075597.348923:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075597.348933:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075597.348944:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923632256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075597.348953:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075597.348956:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075597.348958:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075597.348966:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2286 00000100:00100000:1.0:1769075597.348968:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923632128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075597.348971:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075597.348982:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075597.348986:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075597.349000:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923632384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075597.349007:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075597.349010:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075597.349012:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075597.349022:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2286 00000100:00080000:2.0:1769075597.349032:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075597.349038:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075597.349043:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075597.349049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075597.349055:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923632512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075597.349063:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075597.349066:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075597.349068:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075597.349073:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2286 00000100:00080000:2.0:1769075597.349090:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075597.349095:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075597.349102:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075597.349109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075597/real 1769075597] req@ffff969e0a46f100 x1855007923632128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075597.349113:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923632640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075597.349189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923632128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075597.349202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075597.349208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075597.349231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923632128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075597.349259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075597.349281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075597.349290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075597/real 1769075597] req@ffff969e0a46f800 x1855007923632256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075597.349299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923632256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075597.349305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075597.349308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075597.349310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075597.349315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075597.349326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075597.349332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075597/real 1769075597] req@ffff969e0a46c380 x1855007923632384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075597.349339:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923632384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075597.349344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075597.349346:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075597.349348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075597.349352:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075597.349363:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075597.349376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075597/real 1769075597] req@ffff969e0a46d500 x1855007923632512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075597.349384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923632512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075597.349398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075597.349400:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075597.349403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075597.349407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075597.349420:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075597.349426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075597/real 1769075597] req@ffff969e0a46f480 x1855007923632640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075597.349433:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923632640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075597.349439:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075597.349441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075597.349443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075597.668653:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075598.372796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075598.692664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075599.396784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075599.716641:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075600.420810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075600.740825:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075601.444747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075601.764747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075602.468737:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075602.468753:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075602.468760:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075602.468773:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2291 00000100:00080000:2.0:1769075602.468787:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075602.468798:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075602.468848:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923632768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075602.468872:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075602.468895:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075602.468930:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075602.468936:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075602.468949:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2291 00000100:00080000:2.0:1769075602.468960:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075602.468976:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075602.468989:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923632768:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769075602.469017:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075602.469063:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923632896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075602.469083:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075602.469088:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075602.469092:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075602.469114:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2291 00000100:00080000:2.0:1769075602.469126:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075602.469136:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075602.469147:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075602.469161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075602.469179:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923633024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075602.469210:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075602.469216:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075602.469220:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075602.469231:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2291 00000100:00080000:2.0:1769075602.469239:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075602.469244:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075602.469267:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075602.469276:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075602/real 1769075602] req@ffff969e0a46df80 x1855007923632768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075602.469294:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923633152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075602.469318:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075602.469329:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769075602.469331:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923632768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075602.469338:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075602.469373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075602.469377:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2291 00000100:00080000:1.0:1769075602.469389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769075602.469393:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075602.469402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923632768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075602.469413:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075602.469430:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075602.469453:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923633280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075602.469471:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075602.469516:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075602.469540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075602/real 1769075602] req@ffff969e0a46ea00 x1855007923632896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075602.469558:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923632896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075602.469570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075602.469575:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075602.469579:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923632896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075602.469587:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075602.469652:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075602.469666:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075602/real 1769075602] req@ffff969e0a46ca80 x1855007923633024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075602.469682:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923633024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075602.469712:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075602.469718:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075602.469722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075602.469731:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075602.469760:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075602.469773:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075602/real 1769075602] req@ffff969e0a46d880 x1855007923633152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075602.469797:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923633152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075602.469814:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075602.469821:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075602.469828:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075602.469866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075602.469907:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075602.469925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075602/real 1769075602] req@ffff969e0a46ed80 x1855007923633280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075602.469941:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923633280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075602.469953:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075602.469957:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075602.469971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075602.788707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075603.492900:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075603.812737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075604.516889:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075604.836723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075605.540856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075605.860710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075606.564800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075606.884689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075607.588748:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075607.588762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075607.588784:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075607.588806:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2296 00000100:00080000:2.0:1769075607.588818:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075607.588825:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075607.588884:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923633408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075607.588957:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075607.588976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075607.588999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075607.589002:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075607.589009:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2296 00000100:00080000:2.0:1769075607.589015:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075607.589020:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075607.589032:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075607.589043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923633408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075607.589074:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923633536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075607.589094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075607.589098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075607.589101:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075607.589106:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2296 00000100:00080000:2.0:1769075607.589111:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075607.589116:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075607.589122:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075607.589138:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923633664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075607.589150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075607.589155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075607.589157:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075607.589161:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2296 00000100:00080000:2.0:1769075607.589166:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075607.589173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075607.589179:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075607.589198:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075607.589215:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923633792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075607.589227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075607/real 1769075607] req@ffff969e0a46e300 x1855007923633408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075607.589233:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075607.589258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075607.589264:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075607.589274:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2296 00000100:00080000:2.0:1769075607.589281:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075607.589286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923633408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075607.589293:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075607.589330:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075607.589347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769075607.589351:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923633920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075607.589358:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075607.589372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923633408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075607.589431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075607.589476:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075607.589500:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075607/real 1769075607] req@ffff969e0a46ce00 x1855007923633536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075607.589525:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923633536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075607.589548:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075607.589552:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075607.589568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075607.589577:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075607.589605:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075607.589619:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075607/real 1769075607] req@ffff969e0a46ed80 x1855007923633664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075607.589633:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923633664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075607.589676:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075607.589680:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075607.589694:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075607.589702:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075607.589724:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075607.589736:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075607/real 1769075607] req@ffff969e0a46d880 x1855007923633792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075607.589753:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923633792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075607.589783:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075607.589788:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075607.589803:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075607.589813:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075607.589844:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075607.589878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075607/real 1769075607] req@ffff969e0a46ca80 x1855007923633920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075607.589899:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923633920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075607.589937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075607.589943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075607.589949:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923633920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075607.908704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075608.612742:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075608.932648:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075609.636723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075609.956708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075610.660791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075610.980670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075611.684752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075612.004705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075612.708690:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075612.708702:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075612.708706:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075612.708716:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2301 00000100:00080000:2.0:1769075612.708727:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075612.708734:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075612.708778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075612.708786:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923634048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075612.708834:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075612.708839:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075612.708843:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075612.708859:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2301 00000100:00100000:1.0:1769075612.708883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923634048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075612.708887:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075612.708898:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075612.708909:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075612.708936:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923634176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075612.708948:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075612.708952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075612.708953:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075612.708957:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2301 00000100:00080000:2.0:1769075612.708970:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075612.708975:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075612.708981:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769075612.708984:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075612.708998:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923634304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075612.709008:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075612.709012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075612.709014:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075612.709018:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2301 00000100:00080000:2.0:1769075612.709023:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075612.709024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075612/real 1769075612] req@ffff969e0a46ea00 x1855007923634048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075612.709030:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075612.709040:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075612.709055:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923634432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075612.709057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923634048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075612.709095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075612.709100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075612.709105:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923634048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075612.709127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075612.709132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769075612.709136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634176:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769075612.709138:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075612.709150:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2301 00000100:00080000:2.0:1769075612.709157:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075612.709162:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075612.709167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769075612.709172:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075612.709190:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923634560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075612.709198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075612/real 1769075612] req@ffff969e0a46df80 x1855007923634176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075612.709224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923634176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075612.709236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075612.709241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075612.709246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075612.709255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075612.709284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075612.709298:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075612/real 1769075612] req@ffff969e0a46f480 x1855007923634304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075612.709309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923634304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075612.709316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075612.709319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075612.709321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075612.709327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075612.709343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075612.709350:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075612/real 1769075612] req@ffff969e0a46d500 x1855007923634432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075612.709359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923634432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075612.709374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075612.709377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075612.709380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075612.709397:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075612.709414:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075612.709422:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075612/real 1769075612] req@ffff969e0a46c380 x1855007923634560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075612.709432:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923634560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075612.709438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075612.709441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075612.709443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075613.028686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075613.732823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075614.052701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075614.756836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075615.076740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075615.780843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075616.100670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075616.804794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075617.124658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075617.828674:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075617.828684:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075617.828688:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075617.828696:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2307 00000100:00080000:2.0:1769075617.828706:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075617.828712:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075617.828758:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923634688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075617.828771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075617.828782:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075617.828786:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075617.828788:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075617.828791:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2307 00000100:00080000:2.0:1769075617.828793:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075617.828796:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075617.828817:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075617.828827:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923634688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075617.828830:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923634816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075617.828842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075617.828845:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075617.828846:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075617.828849:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2307 00000100:00080000:2.0:1769075617.828852:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075617.828855:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075617.828858:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075617.828867:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923634944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075617.828873:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075617.828876:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075617.828877:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075617.828879:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2307 00000100:00080000:2.0:1769075617.828881:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075617.828884:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075617.828887:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075617.828895:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923635072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075617.828901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075617.828906:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075617.828906:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075617.828913:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075617.828917:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2307 00000100:00080000:2.0:1769075617.828921:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075617.828924:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075617.828928:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075617.828937:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923635200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075617.828944:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075617/real 1769075617] req@ffff969e0a46f800 x1855007923634688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075617.828962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923634688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075617.828977:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075617.828980:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075617.828983:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923634688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075617.829002:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075617.829020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075617.829029:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075617/real 1769075617] req@ffff969e0a46f100 x1855007923634816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075617.829037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923634816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075617.829052:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075617.829054:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075617.829056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075617.829061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075617.829073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075617.829079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075617/real 1769075617] req@ffff969e0a46c380 x1855007923634944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075617.829086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923634944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075617.829095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075617.829098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075617.829102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923634944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075617.829115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075617.829138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075617.829150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075617/real 1769075617] req@ffff969e0a46d500 x1855007923635072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075617.829161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923635072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075617.829167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075617.829169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075617.829171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075617.829175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075617.829189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075617.829195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075617/real 1769075617] req@ffff969e0a46f480 x1855007923635200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075617.829202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923635200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075617.829208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075617.829210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075617.829212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075618.148670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075618.852782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075619.172670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075619.876877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075620.196668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075620.900769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075621.220752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075621.924740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075622.244686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075622.948712:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075622.948721:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075622.948725:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769075622.948733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075622.948734:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2312 00000100:00080000:0.0:1769075622.948748:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075622.948775:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075622.948811:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923635328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075622.948841:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075622.948845:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075622.948847:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075622.948853:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2312 00000100:00080000:0.0:1769075622.948859:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075622.948864:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075622.948873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923635328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769075622.948875:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075622.948895:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923635456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075622.948908:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075622.948912:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075622.948914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075622.948918:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2312 00000100:00080000:0.0:1769075622.948923:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075622.948927:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075622.948933:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075622.948934:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075622.948951:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923635584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075622.948963:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075622.948964:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075622/real 1769075622] req@ffff969e06a60700 x1855007923635328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075622.948973:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075622.948983:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075622.948985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923635328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075622.948994:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2312 00000100:00080000:1.0:1769075622.948999:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075622.949001:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075622.949005:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075622.949009:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075622.949014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923635328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769075622.949018:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075622.949031:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923635712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075622.949037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635456:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075622.949041:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075622.949045:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075622.949046:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075622.949050:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2312 00000100:00080000:0.0:1769075622.949054:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075622.949060:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075622.949061:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075622.949068:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075622.949077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075622/real 1769075622] req@ffff969e06a60000 x1855007923635456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075622.949080:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923635840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075622.949107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923635456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075622.949115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075622.949118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075622.949121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075622.949126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075622.949138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075622.949144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075622/real 1769075622] req@ffff969e06a63800 x1855007923635584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075622.949160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923635584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075622.949166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075622.949168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075622.949170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075622.949180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075622.949192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075622.949199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075622/real 1769075622] req@ffff969e06a60a80 x1855007923635712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075622.949206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923635712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075622.949220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075622.949222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075622.949225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075622.949228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075622.949239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075622.949245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075622/real 1769075622] req@ffff969e06a61f80 x1855007923635840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075622.949252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923635840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075622.949266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075622.949268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075622.949271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923635840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075623.268709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075623.972773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075624.292677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075624.996787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075625.316689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075626.020786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075626.340674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075627.044704:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075627.364667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075628.068699:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075628.068709:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075628.068713:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075628.068722:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2317 00000100:00080000:3.0:1769075628.068732:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769075628.068738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075628.068739:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075628.068776:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923635968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075628.068798:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075628.068815:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075628.068817:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075628.068821:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2317 00000100:00080000:3.0:1769075628.068824:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075628.068827:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923635968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075628.068830:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075628.068842:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075628.068855:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923636096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075628.068864:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075628.068866:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075628.068868:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075628.068870:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2317 00000100:00080000:3.0:1769075628.068873:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075628.068875:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075628.068878:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075628.068885:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923636224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075628.068891:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075628.068894:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075628.068895:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769075628.068900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075628.068904:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2317 00000100:00080000:3.0:1769075628.068907:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075628.068910:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075628.068919:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075628.068928:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923636352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075628.068932:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075628/real 1769075628] req@ffff969e0a7ffb80 x1855007923635968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075628.068957:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075628.068961:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923635968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075628.068963:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075628.068968:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075628.068973:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075628.068977:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2317 00000100:00080000:1.0:1769075628.068981:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075628.068985:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923635968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075628.068986:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075628.068994:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075628.069000:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075628.069007:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075628.069013:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923636480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075628.069030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075628.069041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075628/real 1769075628] req@ffff969e0a7ff800 x1855007923636096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075628.069065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923636096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075628.069078:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075628.069082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075628.069085:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075628.069090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075628.069106:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075628.069113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075628/real 1769075628] req@ffff969e0a7fdc00 x1855007923636224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075628.069120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923636224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075628.069126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075628.069128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075628.069131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075628.069135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075628.069154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075628.069164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075628/real 1769075628] req@ffff969e0a7fd500 x1855007923636352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075628.069176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923636352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075628.069185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075628.069188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075628.069199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075628.069217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075628.069240:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075628.069251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075628/real 1769075628] req@ffff969e0a7fc000 x1855007923636480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075628.069260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923636480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075628.069266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075628.069268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075628.069270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075628.388678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075629.092870:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075629.412717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075630.116776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075630.436691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075631.140714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075631.460674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075632.164816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075632.484759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075633.188756:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075633.188768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075633.188774:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075633.188787:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2322 00010000:00010000:3.0:1769075633.188793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075633.188800:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075633.188809:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075633.188846:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923636608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075633.188892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075633.188896:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075633.188898:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075633.188904:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2322 00000100:00080000:0.0:1769075633.188910:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075633.188915:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075633.188926:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075633.188938:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923636608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075633.188944:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923636736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075633.188963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075633.188967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075633.188969:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075633.188974:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2322 00000100:00080000:0.0:1769075633.188980:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075633.188984:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075633.188990:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075633.189008:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923636864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075633.189018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075633.189022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075633.189024:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075633.189029:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2322 00000100:00080000:0.0:1769075633.189041:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075633.189046:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075633.189053:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075633.189067:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923636992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075633.189079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075633.189084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075633.189086:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075633.189091:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2322 00000100:00080000:0.0:1769075633.189095:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075633.189098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636736:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075633.189107:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075633.189118:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075633.189136:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923637120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075633.189151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075633.189184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075633.189207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075633.508721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075634.212790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075634.532692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075635.236745:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075635.556693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075636.260758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769075636.260943:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769075636.260999:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769075636.261030:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075636.261085:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075636.261099:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075636.261109:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075636.261122:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769075636.261160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075633/real 1769075636] req@ffff969e06a62680 x1855007923637120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075636.261196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923637120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075636.261217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075636.261225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075636.261230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075636.261240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075633/real 1769075636] req@ffff969e06a62d80 x1855007923636736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075636.261256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923636736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075636.261267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075636.261272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075636.261277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075636.261285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075633/real 1769075636] req@ffff969e06a61500 x1855007923636864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075636.261301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923636864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075636.261312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075636.261317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075636.261350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075636.261360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075633/real 1769075636] req@ffff969e06a61180 x1855007923636992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075636.261378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923636992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075636.261388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075636.261392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075636.261395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923636992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075636.261404:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075633/real 1769075636] req@ffff969e06a60380 x1855007923636608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075636.261419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923636608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075636.261467:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075636.261472:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075636.261477:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923636608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075636.580671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075637.284822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075637.604673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075638.308686:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075638.308704:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075638.308708:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075638.308717:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2327 00000100:00080000:0.0:1769075638.308727:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075638.308734:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075638.308763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075638.308771:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923637248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075638.308803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075638.308826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075638.308828:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075638.308833:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2327 00000100:00100000:1.0:1769075638.308838:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923637248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075638.308840:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075638.308848:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075638.308857:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075638.308878:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923637376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075638.308887:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075638.308890:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075638.308892:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075638.308894:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2327 00000100:00080000:0.0:1769075638.308898:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075638.308900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075638.308904:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075638.308909:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075638.308919:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923637504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075638.308926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075638.308929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075638.308931:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075638.308939:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2327 00000100:00100000:1.0:1769075638.308939:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075638/real 1769075638] req@ffff969e06a61880 x1855007923637248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075638.308944:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075638.308952:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075638.308963:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923637248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075638.308964:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075638.308975:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923637632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075638.308990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075638.308998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075638.309000:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075638.309004:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2327 00000100:00080000:0.0:1769075638.309008:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075638.309008:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075638.309013:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075638.309017:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075638.309019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075638.309026:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923637760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075638.309030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923637248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075638.309072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075638.309101:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075638.309114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075638/real 1769075638] req@ffff969e06a63100 x1855007923637376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075638.309126:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923637376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075638.309134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075638.309137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075638.309140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075638.309147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075638.309167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075638.309177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075638/real 1769075638] req@ffff969e06a60380 x1855007923637504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075638.309188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923637504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075638.309195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075638.309198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075638.309201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075638.309215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075638.309241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075638.309252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075638/real 1769075638] req@ffff969e06a61180 x1855007923637632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075638.309263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923637632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075638.309271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075638.309274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075638.309278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075638.309284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075638.309303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075638.309313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075638/real 1769075638] req@ffff969e06a61500 x1855007923637760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075638.309324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923637760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075638.309332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075638.309335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075638.309338:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923637760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075638.628656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075639.332769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075639.652691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075640.356736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075640.676771:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075641.380774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075641.700652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075642.404727:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075642.724770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075643.428686:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075643.428702:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075643.428709:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075643.428723:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2332 00000100:00080000:0.0:1769075643.428739:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769075643.428746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075643.428752:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075643.428817:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923637888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075643.428850:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075643.428855:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075643.428875:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075643.428884:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2332 00000100:00100000:1.0:1769075643.428884:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923637888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075643.428894:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075643.428900:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075643.428914:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075643.428936:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923638016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075643.428949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075643.428953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075643.428955:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075643.428960:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2332 00000100:00080000:0.0:1769075643.428967:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075643.428972:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075643.428978:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075643.428979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075643.428998:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923638144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075643.429009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075643.429025:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075643.429028:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075643.429031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075643/real 1769075643] req@ffff969e06a62d80 x1855007923637888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075643.429037:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2332 00000100:00080000:0.0:1769075643.429047:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075643.429055:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075643.429056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923637888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075643.429066:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075643.429070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075643.429076:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075643.429081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923637888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075643.429095:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923638272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075643.429109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638016:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075643.429111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075643.429119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075643.429123:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075643.429131:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2332 00000800:00000100:1.0:1769075643.429132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075643.429141:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075643.429146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075643/real 1769075643] req@ffff969e06a62680 x1855007923638016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075643.429151:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075643.429163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923638016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075643.429165:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075643.429177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075643.429180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075643.429184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075643.429189:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923638400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075643.429192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075643.429212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075643.429220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075643/real 1769075643] req@ffff969e06a61f80 x1855007923638144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075643.429239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923638144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075643.429246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075643.429248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075643.429250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075643.429264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075643.429282:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075643.429291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075643/real 1769075643] req@ffff969e06a60a80 x1855007923638272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075643.429301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923638272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075643.429323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075643.429326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075643.429328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075643.429333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075643.429345:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075643.429352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075643/real 1769075643] req@ffff969e06a63800 x1855007923638400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075643.429359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923638400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075643.429383:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075643.429386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075643.429388:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075643.748684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075644.452794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075644.772727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075645.476747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075645.796732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075646.500768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075646.820725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075647.524779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075647.844717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075648.548685:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075648.548698:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075648.548705:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075648.548718:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2337 00000100:00080000:0.0:1769075648.548731:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075648.548740:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075648.548783:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923638528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075648.548786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075648.548824:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075648.548829:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075648.548831:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075648.548837:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2337 00000100:00080000:0.0:1769075648.548844:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075648.548876:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923638528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075648.548877:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075648.548901:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075648.548928:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923638656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075648.548947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075648.548952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075648.548956:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075648.548963:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2337 00000100:00080000:0.0:1769075648.548970:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075648.548976:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075648.548981:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075648.548993:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075648.549014:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923638784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075648.549021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075648/real 1769075648] req@ffff969e06a60000 x1855007923638528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075648.549060:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075648.549066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923638528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075648.549068:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075648.549075:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075648.549084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075648.549088:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2337 00000100:00080000:1.0:1769075648.549094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075648.549100:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075648.549101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923638528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075648.549116:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075648.549125:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075648.549136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075648.549145:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923638912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075648.549156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075648.549181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075648/real 1769075648] req@ffff969e06a60700 x1855007923638656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075648.549188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075648.549193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923638656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075648.549207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075648.549214:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075648.549252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075648.549265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075648.549269:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2337 00000100:00100000:1.0:1769075648.549277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638656:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075648.549280:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075648.549291:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075648.549291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638784:192.168.204.143@tcp:38:kworker.0 00000008:00100000:0.0:1769075648.549315:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075648.549327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075648.549334:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923639040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075648.549346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075648/real 1769075648] req@ffff969e06a63800 x1855007923638784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075648.549360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923638784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075648.549367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075648.549371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075648.549375:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075648.549390:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075648.549403:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075648.549409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075648/real 1769075648] req@ffff969e06a60a80 x1855007923638912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075648.549417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923638912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075648.549422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075648.549424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075648.549432:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923638912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075648.549437:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075648.549450:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075648.549457:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075648/real 1769075648] req@ffff969e06a61f80 x1855007923639040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075648.549465:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923639040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075648.549471:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075648.549473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075648.549474:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075648.868749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075649.572778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075649.892718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075650.596844:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075650.916675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075651.620807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075651.940670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075652.644776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075652.964735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075653.668741:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075653.668757:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075653.668763:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075653.668776:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2342 00000100:00080000:1.0:1769075653.668790:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075653.668799:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075653.668848:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923639168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075653.668877:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075653.668898:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075653.668900:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075653.668905:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2342 00000100:00080000:1.0:1769075653.668909:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075653.668913:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075653.668923:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075653.668934:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923639168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075653.668936:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923639296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075653.668950:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075653.668952:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075653.668954:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075653.668957:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2342 00000100:00080000:1.0:1769075653.668959:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075653.668962:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075653.668965:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075653.668972:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923639424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075653.668978:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075653.668981:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075653.668982:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075653.668985:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2342 00000100:00080000:1.0:1769075653.668987:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075653.668990:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075653.668999:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075653.669007:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923639552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075653.669013:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075653.669016:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075653.669018:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075653.669020:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2342 00000100:00080000:1.0:1769075653.669023:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075653.669026:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075653.669029:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075653.669034:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075653.669039:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923639680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075653.669074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075653/real 1769075653] req@ffff969e10b60700 x1855007923639168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075653.669091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923639168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075653.669101:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075653.669105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075653.669109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923639168:192.168.204.143@tcp:250:kworker.0 00010000:00010000:3.0:1769075653.669120:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075653.669150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075653.669178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075653.669188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075653/real 1769075653] req@ffff969e10b62d80 x1855007923639296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075653.669197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923639296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075653.669203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075653.669205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075653.669208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075653.669212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075653.669223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075653.669229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075653/real 1769075653] req@ffff969e10b60000 x1855007923639424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075653.669236:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923639424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075653.669241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075653.669243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075653.669245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075653.669248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075653.669259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075653.669264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075653/real 1769075653] req@ffff969e10b61180 x1855007923639552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075653.669281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923639552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075653.669287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075653.669290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075653.669292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075653.669295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075653.669308:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075653.669314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075653/real 1769075653] req@ffff969e10b60380 x1855007923639680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075653.669321:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923639680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075653.669327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075653.669329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075653.669330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075653.988734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075654.692749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075655.012759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075655.716826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075656.036676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075656.740801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075657.060740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075657.764816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075658.084726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075658.788771:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075658.788784:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075658.788789:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075658.788822:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2347 00000100:00080000:1.0:1769075658.788838:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075658.788849:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075658.788922:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923639808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075658.789025:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075658.789033:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075658.789037:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075658.789095:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2347 00000100:00080000:1.0:1769075658.789103:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075658.789110:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075658.789126:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075658.789142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923639808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075658.789150:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923639936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075658.789168:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075658.789173:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075658.789176:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075658.789183:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2347 00000100:00080000:1.0:1769075658.789188:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075658.789192:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075658.789197:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075658.789211:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923640064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075658.789222:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075658.789227:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075658.789229:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769075658.789239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769075658.789240:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2347 00000100:00080000:1.0:1769075658.789261:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075658.789268:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075658.789283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075658/real 1769075658] req@ffff969e10b60380 x1855007923639808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769075658.789291:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075658.789302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923639808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075658.789318:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923640192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075658.789363:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075658.789378:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075658.789384:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075658.789394:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2347 00000100:00080000:2.0:1769075658.789395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075658.789403:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075658.789412:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:1.0:1769075658.789420:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075658.789429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923639808:192.168.204.143@tcp:250:kworker.0 00000008:00100000:1.0:1769075658.789431:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075658.789449:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923640320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075658.789474:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075658.789509:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075658.789533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075658/real 1769075658] req@ffff969e10b61180 x1855007923639936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075658.789542:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923639936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075658.789558:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075658.789563:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075658.789566:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923639936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075658.789576:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640064:192.168.204.143@tcp:38:kworker.0 00010000:00010000:3.0:1769075658.789619:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1769075658.789628:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075658.789638:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075658/real 1769075658] req@ffff969e10b60000 x1855007923640064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075658.789647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923640064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075658.789653:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075658.789655:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075658.789658:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075658.789669:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075658.789685:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075658.789692:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075658/real 1769075658] req@ffff969e10b62d80 x1855007923640192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075658.789700:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923640192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075658.789705:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075658.789707:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075658.789709:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075658.789713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075658.789723:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075658.789728:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075658/real 1769075658] req@ffff969e10b60700 x1855007923640320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075658.789735:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923640320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075658.789740:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075658.789742:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075658.789744:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075659.108727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075659.812782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075660.132666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075660.836767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075661.156667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075661.860769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075662.180663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075662.884726:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075663.204662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075663.908756:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075663.908767:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075663.908771:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075663.908782:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2353 00000100:00080000:3.0:1769075663.908796:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075663.908803:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075663.908809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075663.908850:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923640448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075663.908880:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075663.908883:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075663.908884:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075663.908887:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2353 00000100:00080000:3.0:1769075663.908890:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075663.908893:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075663.908923:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075663.908926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923640448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075663.908936:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923640576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075663.908946:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075663.908948:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075663.908950:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075663.908953:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2353 00000100:00080000:3.0:1769075663.908956:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075663.908975:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075663.908981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075663.908992:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923640704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075663.908999:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075663.909002:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075663.909003:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075663.909006:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2353 00000100:00080000:3.0:1769075663.909010:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075663.909013:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075663.909016:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075663.909025:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075663.909027:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923640832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075663.909040:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075663.909044:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075663.909053:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075663.909058:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2353 00000100:00080000:3.0:1769075663.909064:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075663.909070:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075663/real 1769075663] req@ffff969e0a7fca80 x1855007923640448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075663.909071:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075663.909077:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075663.909096:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923640960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075663.909101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923640448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075663.909124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075663.909129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075663.909132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923640448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075663.909157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075663.909175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075663.909183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075663/real 1769075663] req@ffff969e0a7fce00 x1855007923640576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075663.909191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923640576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075663.909198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075663.909200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075663.909202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075663.909205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075663.909215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075663.909221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075663/real 1769075663] req@ffff969e0a7fdf80 x1855007923640704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075663.909228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923640704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075663.909243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075663.909246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075663.909248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075663.909252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075663.909266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075663.909272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075663/real 1769075663] req@ffff969e0a7fc380 x1855007923640832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075663.909279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923640832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075663.909285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075663.909287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075663.909289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075663.909292:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075663.909302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075663.909307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075663/real 1769075663] req@ffff969e0a7fe300 x1855007923640960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075663.909314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923640960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075663.909320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075663.909321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075663.909328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923640960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075664.228691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075664.932777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075665.252713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075665.956812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075666.276726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075666.980772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075667.300688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075668.004764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075668.324704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075669.028704:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075669.028715:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075669.028739:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075669.028749:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2358 00000100:00080000:1.0:1769075669.028760:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075669.028768:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075669.028803:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923641088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075669.028830:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075669.028832:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075669.028834:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075669.028837:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2358 00000100:00080000:1.0:1769075669.028840:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075669.028844:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075669.028852:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075669.028867:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923641088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075669.028868:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923641216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075669.028913:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075669.028917:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075669.028919:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075669.028923:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2358 00000100:00080000:1.0:1769075669.028928:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075669.028948:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075669.028953:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075669.028957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075669.028971:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923641344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075669.028992:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075669/real 1769075669] req@ffff969e10b62680 x1855007923641088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075669.029016:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075669.029036:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075669.029038:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075669.029044:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2358 00000100:00100000:2.0:1769075669.029046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923641088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075669.029048:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075669.029055:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075669.029065:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075669.029071:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769075669.029078:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923641472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075669.029081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075669.029106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923641088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769075669.029110:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075669.029114:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075669.029115:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075669.029118:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2358 00000100:00080000:1.0:1769075669.029121:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075669.029124:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075669.029127:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075669.029131:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075669.029137:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923641600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075669.029171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075669.029186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075669/real 1769075669] req@ffff969e10b62a00 x1855007923641216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075669.029201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923641216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075669.029211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075669.029216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075669.029245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075669.029253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641344:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769075669.029272:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1769075669.029276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075669.029294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075669/real 1769075669] req@ffff969e10b60a80 x1855007923641344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075669.029302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923641344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075669.029324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075669.029326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075669.029329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075669.029333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075669.029348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075669.029354:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075669/real 1769075669] req@ffff969e10b61f80 x1855007923641472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075669.029362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923641472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075669.029368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075669.029370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075669.029372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075669.029384:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075669.029394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075669.029400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075669/real 1769075669] req@ffff969e10b61500 x1855007923641600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075669.029408:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923641600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075669.029422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075669.029424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075669.029426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075669.348769:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075670.052723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075670.372704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075671.076820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075671.396697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075672.100850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075672.420711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075673.124839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075673.444708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075674.148779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075674.148812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075674.148816:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075674.148825:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2363 00010000:00010000:3.0:1769075674.148831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075674.148835:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075674.148842:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075674.148888:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923641728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075674.148915:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075674.148918:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075674.148919:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075674.148922:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2363 00000100:00080000:0.0:1769075674.148925:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075674.148928:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075674.148934:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075674.148943:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923641856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075674.148951:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075674.148953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075674.148955:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075674.148957:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2363 00000100:00080000:0.0:1769075674.148959:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075674.148962:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075674.148965:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075674.148973:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923641984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075674.148973:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923641728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075674.148987:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075674.148989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075674.148990:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075674.148998:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2363 00000100:00080000:0.0:1769075674.149001:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075674.149005:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075674.149009:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075674.149018:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923642112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075674.149024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075674.149026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075674.149027:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075674.149029:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2363 00000100:00080000:0.0:1769075674.149032:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075674.149036:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075674.149037:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075674.149045:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075674.149058:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923642240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075674.149070:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075674/real 1769075674] req@ffff969e06a62680 x1855007923641728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075674.149084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923641728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075674.149092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075674.149096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075674.149099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923641728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075674.149104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075674.149117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075674.149125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075674/real 1769075674] req@ffff969e06a62d80 x1855007923641856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075674.149132:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923641856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075674.149138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075674.149140:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075674.149142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075674.149169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075674.149182:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075674.149189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075674/real 1769075674] req@ffff969e06a61500 x1855007923641984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075674.149197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923641984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075674.149202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075674.149204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075674.149206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923641984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075674.149210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075674.149219:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075674.149233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075674/real 1769075674] req@ffff969e06a61180 x1855007923642112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075674.149241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923642112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075674.149247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075674.149249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075674.149251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075674.149255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075674.149266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075674.149272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075674/real 1769075674] req@ffff969e06a60380 x1855007923642240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075674.149285:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923642240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075674.149291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075674.149293:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075674.149294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075674.468741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075675.172820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075675.492674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075676.196740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075676.516697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075677.220896:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075677.540663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075678.244781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075678.564744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075679.268744:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075679.268755:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075679.268760:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075679.268772:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2368 00000100:00080000:0.0:1769075679.268785:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075679.268793:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075679.268815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075679.268845:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923642368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075679.268871:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075679.268892:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075679.268895:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075679.268900:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2368 00000100:00080000:0.0:1769075679.268904:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075679.268908:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075679.268917:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075679.268928:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923642368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075679.268932:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923642496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075679.268952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075679.268954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075679.268955:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075679.268960:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2368 00000100:00080000:0.0:1769075679.268962:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075679.268964:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075679.268967:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075679.268983:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923642624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075679.268989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075679.268992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075679.268994:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075679.268997:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2368 00000100:00080000:0.0:1769075679.269000:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075679.269003:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075679.269011:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075679.269020:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923642752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075679.269027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075679.269036:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075679.269037:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075679.269043:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075679.269047:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2368 00000100:00080000:0.0:1769075679.269053:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075679.269057:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075679.269061:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075679.269073:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923642880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075679.269087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075679/real 1769075679] req@ffff969e06a63100 x1855007923642368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075679.269104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923642368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075679.269113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075679.269116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075679.269119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923642368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075679.269146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075679.269163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075679.269172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075679/real 1769075679] req@ffff969e06a61880 x1855007923642496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075679.269179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923642496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075679.269203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075679.269206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075679.269208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075679.269213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075679.269226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075679.269233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075679/real 1769075679] req@ffff969e06a60380 x1855007923642624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075679.269240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923642624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075679.269245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075679.269247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075679.269249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075679.269252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075679.269263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075679.269268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075679/real 1769075679] req@ffff969e06a61180 x1855007923642752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075679.269275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923642752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075679.269281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075679.269282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075679.269289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075679.269294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075679.269307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075679.269313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075679/real 1769075679] req@ffff969e06a61500 x1855007923642880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075679.269320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923642880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075679.269326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075679.269327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075679.269329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923642880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075679.588721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075680.292784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075680.612696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075681.316756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075681.636711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075682.340784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075682.660670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075683.364746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075683.684681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075684.388733:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075684.388747:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075684.388754:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075684.388766:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2373 00000100:00080000:0.0:1769075684.388781:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075684.388791:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075684.388841:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923643008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769075684.388885:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075684.388889:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075684.388897:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075684.388900:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075684.388907:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2373 00000100:00080000:0.0:1769075684.388934:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075684.388941:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075684.388954:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075684.388973:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923643136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075684.388990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075684.388995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923643008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075684.388996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075684.389004:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075684.389034:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2373 00000100:00080000:0.0:1769075684.389038:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075684.389042:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075684.389047:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075684.389062:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923643264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075684.389074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075684.389078:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075684.389080:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075684.389083:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2373 00000100:00080000:0.0:1769075684.389087:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075684.389090:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075684.389098:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075684.389145:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923643392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075684.389163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075684.389194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075684.389201:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075684.389206:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075684.389216:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2373 00000100:00080000:0.0:1769075684.389227:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075684.389231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075684/real 1769075684] req@ffff969e06a62d80 x1855007923643008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075684.389241:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075684.389257:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075684.389286:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923643520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075684.389309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923643008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075684.389373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075684.389386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075684.389396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923643008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075684.389458:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075684.389505:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075684.389545:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075684/real 1769075684] req@ffff969e06a62680 x1855007923643136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075684.389573:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923643136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075684.389670:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075684.389679:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075684.389687:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075684.389700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075684.389742:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075684.389766:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075684/real 1769075684] req@ffff969e06a61f80 x1855007923643264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075684.389789:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923643264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075684.389807:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075684.389815:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075684.389822:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075684.389835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075684.389870:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075684.389891:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075684/real 1769075684] req@ffff969e06a60a80 x1855007923643392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075684.389915:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923643392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075684.389948:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075684.389955:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075684.389962:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075684.389975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075684.390033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075684.390050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075684/real 1769075684] req@ffff969e06a63800 x1855007923643520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075684.390067:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923643520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075684.390082:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075684.390087:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075684.390094:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075684.708669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075685.412749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075685.732656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075686.436784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075686.756674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075687.460727:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075687.780683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075688.484780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075688.804680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075689.508707:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075689.508715:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075689.508720:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075689.508728:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2378 00010000:00010000:2.0:1769075689.508735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075689.508738:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075689.508745:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075689.508791:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923643648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075689.508814:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075689.508816:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075689.508818:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075689.508821:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2378 00000100:00080000:3.0:1769075689.508825:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075689.508828:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075689.508836:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075689.508842:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923643648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075689.508846:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923643776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075689.508855:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075689.508857:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075689.508859:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075689.508862:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2378 00000100:00080000:3.0:1769075689.508864:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075689.508866:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075689.508869:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075689.508877:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923643904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075689.508883:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075689.508885:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075689.508886:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075689.508894:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2378 00000100:00080000:3.0:1769075689.508897:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075689.508901:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075689.508902:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075689.508909:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075689.508919:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923644032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075689.508927:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075689.508929:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075689.508930:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075689.508934:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2378 00000100:00100000:2.0:1769075689.508935:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075689/real 1769075689] req@ffff969e0a7fd180 x1855007923643648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075689.508937:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075689.508943:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075689.508950:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075689.508959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923644160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075689.508963:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923643648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075689.508980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075689.508983:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075689.508994:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923643648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075689.509022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075689.509040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075689.509049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075689/real 1769075689] req@ffff969e0a7fea00 x1855007923643776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075689.509057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923643776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075689.509063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075689.509065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075689.509068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075689.509071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075689.509081:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075689.509087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075689/real 1769075689] req@ffff969e0a7fe680 x1855007923643904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075689.509094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923643904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075689.509099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075689.509101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075689.509103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923643904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075689.509106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075689.509116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075689.509129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075689/real 1769075689] req@ffff969e0a7fc700 x1855007923644032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075689.509136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923644032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075689.509142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075689.509143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075689.509145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075689.509149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075689.509160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075689.509166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075689/real 1769075689] req@ffff969e0a7fed80 x1855007923644160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075689.509174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923644160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075689.509179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075689.509181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075689.509183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075689.828703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075690.532965:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075690.852690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075691.556744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075691.876671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075692.580764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075692.900648:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075693.604913:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075693.924650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075694.628753:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075694.628764:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075694.628769:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075694.628779:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2383 00000100:00080000:1.0:1769075694.628790:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075694.628797:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075694.628848:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923644288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075694.628877:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075694.628898:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075694.628900:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075694.628905:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2383 00000100:00080000:1.0:1769075694.628910:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075694.628914:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075694.628932:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075694.628949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923644416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075694.628962:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075694.628969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923644288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769075694.628970:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075694.628980:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075694.628987:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2383 00000100:00080000:1.0:1769075694.628991:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075694.628997:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075694.629006:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075694.629021:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923644544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075694.629033:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075694.629037:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075694.629039:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075694.629043:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2383 00000100:00080000:1.0:1769075694.629048:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075694.629052:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075694.629062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:1.0:1769075694.629072:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075694.629095:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923644672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075694.629108:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075694.629109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075694/real 1769075694] req@ffff969e10b61500 x1855007923644288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075694.629114:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075694.629116:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075694.629125:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2383 00000100:00080000:1.0:1769075694.629138:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075694.629146:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075694.629154:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075694.629154:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923644288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075694.629176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769075694.629184:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923644800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075694.629188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075694.629195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923644288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075694.629236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075694.629269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075694.629293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075694/real 1769075694] req@ffff969e10b61f80 x1855007923644416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075694.629311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923644416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075694.629328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:3.0:1769075694.629328:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075694.629338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075694.629345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075694.629354:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075694.629373:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075694.629383:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075694/real 1769075694] req@ffff969e10b60a80 x1855007923644544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075694.629391:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923644544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075694.629407:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075694.629410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075694.629412:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075694.629417:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075694.629433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075694.629440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075694/real 1769075694] req@ffff969e10b62a00 x1855007923644672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075694.629475:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923644672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075694.629481:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075694.629484:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075694.629486:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075694.629491:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075694.629506:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075694.629514:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075694/real 1769075694] req@ffff969e10b62680 x1855007923644800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075694.629521:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923644800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075694.629527:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075694.629529:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075694.629536:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923644800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075694.948678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075695.652783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075695.972678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075696.676828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075696.996711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075697.700722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075698.020683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075698.724735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075699.044665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075699.748706:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075699.748715:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075699.748720:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769075699.748721:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075699.748731:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2388 00000100:00080000:3.0:1769075699.748740:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075699.748762:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075699.748790:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923644928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075699.748813:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075699.748815:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075699.748817:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075699.748820:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2388 00000100:00080000:3.0:1769075699.748823:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075699.748827:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075699.748834:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075699.748842:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923644928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075699.748847:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923645056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075699.748860:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075699.748862:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075699.748863:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075699.748865:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2388 00000100:00080000:3.0:1769075699.748869:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075699.748871:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075699.748874:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075699.748881:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923645184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075699.748888:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075699.748890:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075699.748896:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075699.748899:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2388 00000100:00080000:3.0:1769075699.748903:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075699.748906:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075699.748910:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075699.748919:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923645312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075699.748926:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075699.748928:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075699.748929:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075699.748932:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2388 00000100:00080000:3.0:1769075699.748934:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075699.748936:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075699.748946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645056:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075699.748947:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075699.748959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923645440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075699.748984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075699.749007:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075699.749029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075700.068694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075700.772765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075701.092643:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075701.796781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075702.116653:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000008:00020000:3.0:1769075702.820684:0:15142:0:(osc_cache.c:945:osc_extent_wait()) extent ffff969e03826e40@{[0 -> 0/1023], [3|0|+|rpc|wihY|ffff969e10aa3c00], [28672|1|+|-|ffff969e05654800|1024|ffff969e11070000]} lustre-OST0001-osc-ffff969e12057000: wait ext to 0 timedout, recovery in progress? 00010000:00010000:2.0:1769075702.820784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:0.0:1769075702.820872:0:15141:0:(osc_cache.c:945:osc_extent_wait()) extent ffff969e03826a80@{[0 -> 0/1023], [3|0|+|rpc|wihY|ffff969e10aa3a40], [28672|1|+|-|ffff969e05655000|1024|ffff969e098acf80]} lustre-OST0000-osc-ffff969e12057000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:0.0:1769075702.820886:0:15141:0:(osc_cache.c:945:osc_extent_wait()) ### extent: 00000000e7c05bc1 ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e05655000/0x44e2946041b60907 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88b2e9 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000400:00000100:1.0:1769075702.829374:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:1.0:1769075702.829409:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769075702.829425:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769075702.829454:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769075702.829461:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769075702.829467:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769075702.829491:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075702.829515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075699/real 1769075702] req@ffff969e0a7fed80 x1855007923645056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075702.829536:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923645056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075702.829546:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075702.829550:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075702.829553:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075702.829558:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075699/real 1769075702] req@ffff969e0a7fc700 x1855007923645184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075702.829566:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923645184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075702.829608:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075702.829611:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075702.829614:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075702.829623:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075699/real 1769075702] req@ffff969e0a7fe680 x1855007923645312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075702.829638:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923645312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075702.829647:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075702.829652:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075702.829656:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075702.829663:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075699/real 1769075702] req@ffff969e0a7fd880 x1855007923644928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075702.829676:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923644928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075702.829683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075702.829686:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075702.829688:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923644928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075702.829708:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075699/real 1769075702] req@ffff969e0a7fea00 x1855007923645440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075702.829716:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923645440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075702.829727:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075702.829729:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075702.829732:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645440:192.168.204.143@tcp:8:kworker.0 00000008:00020000:3.0:1769075702.845682:0:15142:0:(osc_cache.c:945:osc_extent_wait()) ### extent: 00000000e198f9f8 ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e05654800/0x44e2946041b6091c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88b321 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000100:00100000:1.0:1769075703.141670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075703.844831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075704.164667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075704.868744:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075704.868756:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075704.868762:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075704.868774:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2394 00000100:00080000:3.0:1769075704.868818:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075704.868830:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075704.868842:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075704.868892:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923645568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075704.868931:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075704.868946:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075704.868948:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075704.868954:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2394 00000100:00080000:3.0:1769075704.868960:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075704.868967:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075704.868979:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075704.868985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923645568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075704.869001:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923645696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075704.869027:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075704.869033:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075704.869037:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075704.869044:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2394 00000100:00080000:3.0:1769075704.869049:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075704.869056:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075704.869065:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075704.869097:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923645824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075704.869115:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075704.869120:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075704.869150:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075704.869155:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075704.869164:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2394 00000100:00080000:3.0:1769075704.869174:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075704.869183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075704/real 1769075704] req@ffff969e0a7fd180 x1855007923645568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075704.869188:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075704.869214:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075704.869238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923645568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075704.869241:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923645952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075704.869320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075704.869331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075704.869354:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923645568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075704.869363:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075704.869372:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075704.869377:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075704.869386:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2394 00000100:00080000:3.0:1769075704.869394:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075704.869401:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645696:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075704.869408:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075704.869423:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075704.869449:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923646080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075704.869467:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075704.869502:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075704/real 1769075704] req@ffff969e0a7fe300 x1855007923645696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075704.869538:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923645696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075704.869552:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075704.869572:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075704.869579:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075704.869635:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075704.869689:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075704.869711:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075704/real 1769075704] req@ffff969e0a7fc380 x1855007923645824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075704.869733:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923645824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075704.869747:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075704.869752:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075704.869758:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075704.869768:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075704.869803:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075704.869822:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075704/real 1769075704] req@ffff969e0a7fdf80 x1855007923645952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075704.869842:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923645952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075704.869858:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075704.869863:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075704.869869:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923645952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075704.869900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075704.869955:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075704.869976:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075704/real 1769075704] req@ffff969e0a7fce00 x1855007923646080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075704.869996:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923646080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075704.870011:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075704.870017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075704.870022:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075705.188747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075705.892882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075706.212709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075706.916789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075707.236712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075707.940839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075708.260746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075708.964816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075709.284733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075709.988684:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075709.988692:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075709.988696:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769075709.988701:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075709.988704:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2399 00000100:00080000:0.0:1769075709.988715:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075709.988721:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075709.988751:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923646208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075709.988772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075709.988787:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075709.988788:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075709.988792:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2399 00000100:00080000:0.0:1769075709.988795:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075709.988799:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075709.988806:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075709.988817:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923646336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075709.988822:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923646208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075709.988833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075709.988835:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075709.988837:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075709.988840:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2399 00000100:00080000:0.0:1769075709.988843:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075709.988845:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075709.988848:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075709.988858:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923646464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075709.988865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075709.988867:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075709.988869:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075709.988871:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2399 00000100:00080000:0.0:1769075709.988874:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075709.988876:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075709.988885:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075709.988887:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075709.988898:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923646592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075709.988906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075709.988908:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075709.988910:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075709.988913:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2399 00000100:00080000:0.0:1769075709.988917:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075709.988917:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075709/real 1769075709] req@ffff969e06a60700 x1855007923646208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075709.988924:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075709.988930:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075709.988940:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923646720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075709.988942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923646208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075709.988964:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075709.988968:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075709.988971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923646208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075709.988989:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075709.989003:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075709.989010:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075709/real 1769075709] req@ffff969e06a60000 x1855007923646336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075709.989030:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923646336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075709.989036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075709.989038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075709.989041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075709.989045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075709.989068:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075709.989074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075709/real 1769075709] req@ffff969e06a63800 x1855007923646464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075709.989082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923646464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075709.989087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075709.989089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075709.989090:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075709.989094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075709.989105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075709.989110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075709/real 1769075709] req@ffff969e06a60a80 x1855007923646592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075709.989127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923646592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075709.989141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075709.989143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075709.989146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075709.989149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075709.989161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075709.989167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075709/real 1769075709] req@ffff969e06a61f80 x1855007923646720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075709.989174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923646720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075709.989179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075709.989181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075709.989183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075710.308653:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075711.012729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075711.332678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075712.036872:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075712.356670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075713.060910:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075713.380749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075714.084789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075714.404674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075715.108704:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075715.108714:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075715.108719:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075715.108730:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2404 00000100:00080000:2.0:1769075715.108741:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769075715.108748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075715.108750:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075715.108791:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923646848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075715.108817:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075715.108836:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075715.108837:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075715.108842:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2404 00000100:00080000:2.0:1769075715.108845:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075715.108848:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075715.108856:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075715.108861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923646848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075715.108871:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923646976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075715.108882:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075715.108886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075715.108888:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075715.108893:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2404 00000100:00080000:2.0:1769075715.108896:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075715.108899:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075715.108902:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075715.108910:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923647104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075715.108917:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075715.108920:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075715.108922:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075715.108924:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2404 00000100:00080000:2.0:1769075715.108927:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075715.108928:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075715.108938:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075715.108947:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923647232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075715.108958:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075715.108970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075715.108974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075715.108976:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075715.108980:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2404 00000100:00080000:2.0:1769075715.108984:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075715.108988:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075715.108992:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075715.109004:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923647360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075715.109013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075715/real 1769075715] req@ffff969e0a46df80 x1855007923646848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075715.109036:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923646848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075715.109045:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075715.109049:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075715.109053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923646848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075715.109080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075715.109097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075715.109122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075715/real 1769075715] req@ffff969e0a46ea00 x1855007923646976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075715.109160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923646976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075715.109167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075715.109170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075715.109172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923646976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075715.109177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075715.109193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075715.109200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075715/real 1769075715] req@ffff969e0a46ca80 x1855007923647104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075715.109208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923647104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075715.109214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075715.109216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075715.109217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075715.109221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075715.109233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075715.109238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075715/real 1769075715] req@ffff969e0a46d880 x1855007923647232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075715.109246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923647232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075715.109262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075715.109264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075715.109267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075715.109271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075715.109297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075715.109305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075715/real 1769075715] req@ffff969e0a46ed80 x1855007923647360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075715.109313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923647360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075715.109319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075715.109321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075715.109324:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075715.428791:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075716.132808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075716.452661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075717.156796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075717.476696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075718.180751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075718.500652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075719.204843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075719.524673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075720.228703:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075720.228726:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075720.228731:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075720.228741:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2409 00000100:00080000:0.0:1769075720.228751:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075720.228770:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075720.228775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075720.228812:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923647488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075720.228833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075720.228837:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075720.228840:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075720.228851:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2409 00000100:00080000:0.0:1769075720.228869:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075720.228873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923647488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075720.228874:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075720.228883:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075720.228895:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923647616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075720.228906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075720.228909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075720.228911:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075720.228914:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2409 00000100:00080000:0.0:1769075720.228919:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075720.228923:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075720.228927:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075720.228933:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075720.228935:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923647744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075720.228944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075720.228947:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075720.228948:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075720.228952:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2409 00000100:00080000:0.0:1769075720.228954:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075720.228958:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075720.228962:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075720.228963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075720/real 1769075720] req@ffff969e06a62680 x1855007923647488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075720.228972:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923647872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075720.228980:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923647488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075720.228994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075720.228996:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075720.228998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075720.228999:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075720.229001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075720.229003:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2409 00000100:00100000:1.0:1769075720.229005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923647488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075720.229007:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075720.229011:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075720.229034:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075720.229046:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923648000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075720.229058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075720.229084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075720.229097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075720/real 1769075720] req@ffff969e06a62d80 x1855007923647616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075720.229106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923647616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075720.229128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075720.229130:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075720.229133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075720.229137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075720.229151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075720.229158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075720/real 1769075720] req@ffff969e06a61500 x1855007923647744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075720.229165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923647744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075720.229172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075720.229190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075720.229193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075720.229209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075720.229231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075720.229244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075720/real 1769075720] req@ffff969e06a61180 x1855007923647872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075720.229257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923647872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075720.229267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075720.229270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075720.229274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923647872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075720.229280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075720.229302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075720.229313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075720/real 1769075720] req@ffff969e06a60380 x1855007923648000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075720.229325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923648000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075720.229331:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075720.229333:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075720.229336:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075720.548648:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075721.252741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075721.572706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075722.276829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075722.596780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075723.300773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075723.620757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075724.324805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075724.644738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075725.348756:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075725.348766:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075725.348770:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075725.348779:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2414 00010000:00010000:2.0:1769075725.348784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075725.348790:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075725.348799:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075725.348836:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923648128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075725.348862:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075725.348865:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075725.348867:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075725.348870:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2414 00000100:00080000:3.0:1769075725.348873:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075725.348876:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075725.348883:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075725.348894:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923648256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075725.348907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923648128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075725.348924:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075725.348945:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075725.348948:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075725.348952:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2414 00000100:00080000:3.0:1769075725.348956:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075725.348960:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075725.348965:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075725.348978:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923648384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075725.348986:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075725.348989:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075725.348991:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075725.348994:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2414 00000100:00080000:3.0:1769075725.348997:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075725.349021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075725.349022:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075725.349031:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075725.349049:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923648512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075725.349065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075725/real 1769075725] req@ffff969e0a7fca80 x1855007923648128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075725.349066:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075725.349097:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075725.349105:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075725.349116:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2414 00000100:00100000:1.0:1769075725.349120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923648128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075725.349137:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075725.349140:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075725.349152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075725.349152:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075725.349161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923648128:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075725.349167:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075725.349186:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923648640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075725.349193:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075725.349237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075725.349258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075725/real 1769075725] req@ffff969e0a7fc000 x1855007923648256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075725.349278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923648256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075725.349293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075725.349299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075725.349304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075725.349314:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075725.349353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075725.349367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075725/real 1769075725] req@ffff969e0a7fd500 x1855007923648384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075725.349397:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923648384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075725.349409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075725.349413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075725.349418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075725.349426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075725.349448:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075725.349458:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075725/real 1769075725] req@ffff969e0a7fdc00 x1855007923648512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075725.349468:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923648512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075725.349475:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075725.349477:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075725.349480:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075725.349493:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075725.349506:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075725.349513:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075725/real 1769075725] req@ffff969e0a7ff800 x1855007923648640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075725.349522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923648640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075725.349539:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075725.349542:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075725.349545:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075725.668669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075726.372901:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075726.692700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075727.396841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075727.716649:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075728.420842:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075728.740711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075729.444836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075729.764687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075730.468787:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075730.468800:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075730.468804:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075730.468815:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2419 00000100:00080000:3.0:1769075730.468831:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075730.468873:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075730.468883:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075730.468918:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923648768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075730.468955:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075730.468959:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075730.468960:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075730.468964:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2419 00000100:00080000:3.0:1769075730.468968:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075730.468972:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075730.468981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075730.469021:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923648896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075730.469031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923648768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075730.469031:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075730.469053:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075730.469054:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075730.469059:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2419 00000100:00080000:3.0:1769075730.469062:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075730.469082:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075730.469090:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075730.469108:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923649024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075730.469117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075730.469129:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075730.469135:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075730.469139:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075730.469146:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2419 00000100:00080000:3.0:1769075730.469153:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075730.469159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075730/real 1769075730] req@ffff969e0a7ffb80 x1855007923648768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075730.469162:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075730.469190:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075730.469201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923648768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075730.469207:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923649152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075730.469212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075730.469221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075730.469227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923648768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075730.469228:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075730.469236:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075730.469238:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075730.469243:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2419 00000100:00080000:3.0:1769075730.469248:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075730.469251:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648896:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075730.469257:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075730.469269:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075730.469275:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075730.469301:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923649280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075730.469311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075730/real 1769075730] req@ffff969e0a7ff800 x1855007923648896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075730.469346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923648896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075730.469365:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075730.469368:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075730.469372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923648896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075730.469377:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075730.469396:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075730.469405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075730/real 1769075730] req@ffff969e0a7fdc00 x1855007923649024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075730.469415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923649024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075730.469422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075730.469424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075730.469427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075730.469431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075730.469463:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075730.469477:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075730/real 1769075730] req@ffff969e0a7fd500 x1855007923649152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075730.469494:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923649152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075730.469505:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075730.469509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075730.469523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075730.469554:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075730.469575:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075730.469585:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075730/real 1769075730] req@ffff969e0a7fc000 x1855007923649280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075730.469595:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923649280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075730.469603:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075730.469606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075730.469609:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075730.788736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075731.492806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075731.812707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075732.516941:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075732.836688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075733.540848:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075733.860644:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075734.564858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075734.884750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075735.588807:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075735.588822:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075735.588829:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075735.588844:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2424 00000100:00080000:3.0:1769075735.588859:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075735.588870:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075735.588935:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075735.588939:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923649408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075735.588978:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075735.588982:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075735.588983:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075735.588987:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2424 00000100:00080000:3.0:1769075735.588992:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075735.588996:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075735.589012:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075735.589017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923649408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075735.589029:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923649536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075735.589040:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075735.589043:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075735.589044:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075735.589048:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2424 00000100:00080000:3.0:1769075735.589082:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075735.589086:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075735.589091:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075735.589092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075735.589107:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923649664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075735.589117:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075735.589119:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075735.589121:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075735.589126:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2424 00000100:00080000:3.0:1769075735.589130:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075735.589150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075735/real 1769075735] req@ffff969e0a7fca80 x1855007923649408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075735.589156:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075735.589170:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075735.589182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923649408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075735.589193:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923649792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075735.589198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075735.589226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075735.589237:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075735.589249:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769075735.589252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923649408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075735.589254:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075735.589266:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2424 00000100:00080000:3.0:1769075735.589273:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075735.589278:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075735.589296:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075735.589316:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923649920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075735.589331:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075735.589380:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075735.589403:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075735/real 1769075735] req@ffff969e0a7fce00 x1855007923649536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075735.589436:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923649536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075735.589445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075735.589449:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075735.589452:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075735.589458:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075735.589474:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075735.589499:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075735/real 1769075735] req@ffff969e0a7fdf80 x1855007923649664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075735.589512:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923649664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075735.589521:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075735.589524:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075735.589527:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075735.589533:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075735.589549:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075735.589580:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075735/real 1769075735] req@ffff969e0a7fc380 x1855007923649792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075735.589592:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923649792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075735.589601:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075735.589604:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075735.589607:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075735.589677:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075735.589693:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075735.589701:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075735/real 1769075735] req@ffff969e0a7fe300 x1855007923649920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075735.589710:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923649920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075735.589717:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075735.589719:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075735.589721:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923649920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075735.908683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075736.612874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075736.932674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075737.636828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075737.956780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075738.660824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075738.980757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075739.684862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075740.004748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075740.708774:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075740.708786:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075740.708792:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075740.708805:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2429 00010000:00010000:2.0:1769075740.708818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075740.708819:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075740.708833:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075740.708877:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923650048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075740.708912:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075740.708916:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075740.708918:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075740.708923:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2429 00000100:00080000:3.0:1769075740.708927:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075740.708931:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075740.708942:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075740.708957:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923650176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075740.708969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923650048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075740.708975:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075740.708982:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075740.709003:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075740.709010:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2429 00000100:00080000:3.0:1769075740.709016:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075740.709030:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075740.709037:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075740.709053:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923650304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075740.709065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075740.709072:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075740.709076:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075740.709081:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075740.709089:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2429 00000100:00080000:3.0:1769075740.709094:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075740.709099:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075740.709105:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075740.709120:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923650432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075740.709133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075740/real 1769075740] req@ffff969e0a7fd180 x1855007923650048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075740.709135:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075740.709146:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075740.709152:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075740.709164:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2429 00000100:00080000:3.0:1769075740.709170:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075740.709175:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075740.709180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923650048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075740.709184:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075740.709195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075740.709203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075740.709215:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923650560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075740.709216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923650048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075740.709258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075740.709307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075740.709321:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075740/real 1769075740] req@ffff969e0a7fea00 x1855007923650176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075740.709336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923650176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075740.709346:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075740.709350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075740.709354:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075740.709360:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075740.709377:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075740.709387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075740/real 1769075740] req@ffff969e0a7fd880 x1855007923650304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075740.709400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923650304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075740.709427:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075740.709432:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075740.709436:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075740.709443:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075740.709466:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075740.709484:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075740/real 1769075740] req@ffff969e0a7fe680 x1855007923650432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075740.709497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923650432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075740.709506:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075740.709509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075740.709512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075740.709519:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075740.709535:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075740.709566:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075740/real 1769075740] req@ffff969e0a7fc700 x1855007923650560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075740.709578:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923650560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075740.709586:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075740.709589:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075740.709600:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075741.028775:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075741.732775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075742.052681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075742.756799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075743.076701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075743.780760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075744.100736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075744.804777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075745.124716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075745.828963:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075745.828970:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075745.828981:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075745.828987:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075745.829001:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2435 00000100:00080000:0.0:1769075745.829015:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075745.829025:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075745.829072:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923650688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075745.829112:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075745.829117:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075745.829120:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075745.829127:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2435 00000100:00080000:0.0:1769075745.829133:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075745.829141:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075745.829166:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075745.829176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923650688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075745.829207:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923650816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075745.829236:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075745.829242:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075745.829245:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075745.829252:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2435 00000100:00080000:0.0:1769075745.829260:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075745.829270:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075745.829272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075745.829283:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075745.829306:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923650944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075745.829322:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075745.829328:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075745/real 1769075745] req@ffff969e06a61880 x1855007923650688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075745.829330:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075745.829336:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075745.829350:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2435 00000100:00080000:0.0:1769075745.829366:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075745.829369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923650688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075745.829376:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075745.829390:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075745.829398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075745.829410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075745.829413:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923651072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075745.829423:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923650688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075745.829445:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075745.829451:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075745.829464:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075745.829470:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650816:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075745.829476:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2435 00000100:00080000:0.0:1769075745.829484:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075745.829490:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075745.829498:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075745.829538:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923651200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075745.829558:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075745.829581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075745/real 1769075745] req@ffff969e06a63100 x1855007923650816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075745.829669:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923650816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075745.829677:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075745.829680:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075745.829691:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075745.829697:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075745.829721:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075745.829731:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075745/real 1769075745] req@ffff969e06a60380 x1855007923650944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075745.829742:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923650944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075745.829748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075745.829751:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075745.829754:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923650944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075745.829758:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075745.829772:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075745.829779:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075745/real 1769075745] req@ffff969e06a61180 x1855007923651072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075745.829788:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923651072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075745.829795:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075745.829797:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075745.829800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075745.829815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075745.829837:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075745.829866:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075745/real 1769075745] req@ffff969e06a61500 x1855007923651200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075745.829884:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923651200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075745.829894:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075745.829906:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075745.829909:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075746.148719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075746.852860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075747.172710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075747.876801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075748.196709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075748.900799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075749.220710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075749.924778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075750.244684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075750.948759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075750.948770:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075750.948786:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769075750.948789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075750.948796:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2440 00000100:00080000:2.0:1769075750.948809:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075750.948816:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075750.948854:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923651328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075750.948884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075750.948887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075750.948889:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075750.948893:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2440 00000100:00080000:2.0:1769075750.948896:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075750.948899:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075750.948908:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075750.948920:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923651456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075750.948931:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075750.948934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075750.948936:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075750.948939:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2440 00000100:00080000:2.0:1769075750.948941:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075750.948948:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075750.948951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923651328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769075750.948972:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075750.948988:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923651584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075750.948999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075750.949002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075750.949004:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075750.949008:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2440 00000100:00080000:2.0:1769075750.949021:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075750.949025:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075750.949033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769075750.949033:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075750.949049:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923651712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075750.949059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075750.949063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075750.949065:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075750.949068:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2440 00000100:00080000:2.0:1769075750.949072:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075750.949076:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075750.949090:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075750.949091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075750/real 1769075750] req@ffff969e0a46ce00 x1855007923651328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075750.949104:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923651840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075750.949137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923651328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075750.949149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075750.949153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075750.949157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923651328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075750.949166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075750.949191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075750.949215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075750/real 1769075750] req@ffff969e0a46e300 x1855007923651456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075750.949227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923651456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075750.949235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075750.949238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075750.949241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075750.949267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075750.949285:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075750.949294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075750/real 1769075750] req@ffff969e0a46ed80 x1855007923651584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075750.949304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923651584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075750.949311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075750.949314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075750.949316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075750.949322:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075750.949336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075750.949343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075750/real 1769075750] req@ffff969e0a46d880 x1855007923651712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075750.949353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923651712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075750.949369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075750.949372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075750.949375:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075750.949380:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075750.949398:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075750.949407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075750/real 1769075750] req@ffff969e0a46ca80 x1855007923651840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075750.949417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923651840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075750.949425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075750.949427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075750.949430:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923651840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075751.268718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075751.972696:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075752.292870:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075752.996766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075753.316664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075754.020806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075754.340677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075755.044748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075755.364664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075756.068678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075756.068687:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075756.068707:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075756.068716:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2445 00000100:00080000:2.0:1769075756.068725:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075756.068731:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769075756.068743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075756.068762:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923651968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075756.068784:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075756.068788:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075756.068790:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075756.068795:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2445 00000100:00080000:2.0:1769075756.068800:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075756.068805:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075756.068814:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075756.068818:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923651968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075756.068832:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923652096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075756.068845:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075756.068849:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075756.068851:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075756.068855:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2445 00000100:00080000:2.0:1769075756.068860:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075756.068863:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075756.068868:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075756.068877:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075756.068881:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923652224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075756.068899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075756.068902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075756.068903:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075756.068909:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075756/real 1769075756] req@ffff969e0a46ea00 x1855007923651968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075756.068909:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2445 00000100:00080000:2.0:1769075756.068916:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075756.068923:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075756.068930:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075756.068935:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923651968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075756.068941:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923652352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075756.068962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075756.068966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075756.068968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075756.068969:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075756.068976:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769075756.068977:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2445 00000100:00080000:2.0:1769075756.068984:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075756.068986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923651968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075756.068990:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075756.068995:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075756.069001:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923652480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075756.069022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075756.069042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075756.069051:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075756/real 1769075756] req@ffff969e0a46df80 x1855007923652096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075756.069060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923652096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075756.069066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075756.069068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075756.069071:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075756.069074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075756.069084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075756.069090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075756/real 1769075756] req@ffff969e0a46f480 x1855007923652224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075756.069097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923652224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075756.069102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075756.069104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075756.069106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075756.069116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075756.069130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075756.069136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075756/real 1769075756] req@ffff969e0a46d500 x1855007923652352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075756.069143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923652352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075756.069149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075756.069151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075756.069153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075756.069156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075756.069166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075756.069171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075756/real 1769075756] req@ffff969e0a46c380 x1855007923652480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075756.069178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923652480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075756.069184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075756.069185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075756.069187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075756.388656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075757.092894:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075757.412654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075758.116780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075758.436692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075759.140755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075759.460649:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769075760.164849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075760.484733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075761.188680:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075761.188690:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075761.188694:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075761.188702:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2450 00000100:00080000:3.0:1769075761.188712:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075761.188718:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075761.188725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075761.188752:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923652608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075761.188775:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075761.188778:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075761.188779:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075761.188782:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2450 00000100:00080000:3.0:1769075761.188800:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075761.188804:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075761.188812:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075761.188813:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923652608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075761.188829:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923652736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075761.188843:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075761.188847:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075761.188849:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075761.188854:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2450 00000100:00080000:3.0:1769075761.188859:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075761.188864:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075761.188870:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075761.188874:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075761.188889:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923652864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075761.188899:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075761.188901:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075761.188903:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075761.188907:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075761/real 1769075761] req@ffff969e0a7fed80 x1855007923652608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075761.188910:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2450 00000100:00080000:3.0:1769075761.188917:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075761.188928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923652608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075761.188933:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075761.188943:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075761.188946:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075761.188951:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075761.188957:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923652992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075761.188958:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923652608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075761.188984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652736:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075761.188984:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075761.188992:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075761.188995:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075761.188999:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2450 00000100:00080000:3.0:1769075761.189005:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075761.189009:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075761.189013:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075761.189017:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075761.189027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075761/real 1769075761] req@ffff969e0a7fc700 x1855007923652736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075761.189031:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923653120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075761.189054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923652736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075761.189061:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075761.189065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075761.189068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075761.189073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075761.189085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075761.189091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075761/real 1769075761] req@ffff969e0a7fe680 x1855007923652864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075761.189109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923652864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075761.189115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075761.189117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075761.189119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075761.189128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075761.189141:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075761.189148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075761/real 1769075761] req@ffff969e0a7fd880 x1855007923652992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075761.189156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923652992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075761.189161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075761.189163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075761.189165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923652992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075761.189168:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075761.189178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075761.189183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075761/real 1769075761] req@ffff969e0a7fea00 x1855007923653120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075761.189190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923653120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075761.189205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075761.189207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075761.189210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075761.508689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075762.212745:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075762.532704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075763.236770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075763.556697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075764.260756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075764.580676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075765.284727:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075765.604664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075766.308672:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075766.308684:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075766.308689:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075766.308700:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2455 00000100:00080000:3.0:1769075766.308712:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075766.308720:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075766.308764:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923653248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075766.308769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075766.308795:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075766.308799:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075766.308800:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075766.308805:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2455 00000100:00080000:3.0:1769075766.308807:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075766.308811:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075766.308837:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075766.308838:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923653248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075766.308852:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923653376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075766.308861:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075766.308864:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075766.308866:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075766.308868:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2455 00000100:00080000:3.0:1769075766.308872:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075766.308875:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075766.308878:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075766.308887:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923653504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075766.308894:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075766.308897:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075766.308899:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075766.308901:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2455 00000100:00080000:3.0:1769075766.308903:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075766.308905:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075766.308908:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075766.308928:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923653632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075766.308935:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075766.308938:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075766.308944:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075766.308947:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2455 00000100:00080000:3.0:1769075766.308953:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075766.308957:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075766.308962:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075766.308974:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075766.308974:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923653760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075766.309024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075766.309048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075766.309068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075766.628688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075767.332789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075767.652690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075768.356831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075768.676742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075769.380802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769075769.380883:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769075769.380935:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769075769.380958:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075769.380997:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075769.381007:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075769.381016:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075769.381024:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769075769.381040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075766/real 1769075769] req@ffff969e0a7fe300 x1855007923653376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.381066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923653376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.381077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.381082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075769.381103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075769.381109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075766/real 1769075769] req@ffff969e0a7fc380 x1855007923653504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.381120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923653504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.381125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.381128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075769.381130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075769.381136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075766/real 1769075769] req@ffff969e0a7fdf80 x1855007923653632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.381144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923653632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.381150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.381153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075769.381157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075769.381164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075766/real 1769075769] req@ffff969e0a7fd180 x1855007923653248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.381178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923653248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.381186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.381209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075769.381224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923653248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075769.381259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075766/real 1769075769] req@ffff969e0a7fce00 x1855007923653760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075769.381274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923653760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075769.381284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075769.381289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075769.381292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923653760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075769.700701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075770.404776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075770.724705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075771.428752:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075771.428766:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075771.428773:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769075771.428773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075771.428796:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2460 00000100:00080000:1.0:1769075771.428811:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075771.428820:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075771.428866:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923653888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075771.428893:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075771.428896:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075771.428898:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075771.428902:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2460 00000100:00080000:1.0:1769075771.428905:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075771.428927:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075771.428940:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075771.428951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923653888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075771.428962:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923654016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075771.428983:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075771.428987:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075771.428990:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075771.428996:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2460 00000100:00080000:1.0:1769075771.429003:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075771.429009:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075771.429016:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075771.429030:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923654144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075771.429042:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769075771.429051:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769075771.429053:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075771.429058:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075771.429066:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2460 00000100:00080000:1.0:1769075771.429073:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075771.429079:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075771.429086:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075771.429115:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923654272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075771.429117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075771/real 1769075771] req@ffff969e10b60700 x1855007923653888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075771.429201:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075771.429205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923653888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075771.429226:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075771.429236:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075771.429243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075771.429245:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2460 00000100:00080000:1.0:1769075771.429260:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075771.429270:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769075771.429279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:1.0:1769075771.429284:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075771.429293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923653888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075771.429313:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923654400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075771.429343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075771.429396:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075771.429416:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075771/real 1769075771] req@ffff969e10b62d80 x1855007923654016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075771.429433:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923654016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075771.429473:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075771.429479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075771.429485:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075771.429496:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075771.429530:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075771.429550:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075771/real 1769075771] req@ffff969e10b60000 x1855007923654144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075771.429565:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923654144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075771.429576:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075771.429581:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075771.429585:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075771.429640:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075771.429669:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075771.429693:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075771/real 1769075771] req@ffff969e10b61180 x1855007923654272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075771.429710:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923654272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075771.429723:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075771.429732:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075771.429747:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075771.429757:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075771.429785:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075771.429799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075771/real 1769075771] req@ffff969e10b60380 x1855007923654400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075771.429813:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923654400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075771.429825:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075771.429829:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075771.429833:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075771.748691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075772.452811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075772.772672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075773.476753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075773.796685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075774.500819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075774.820685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075775.524725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075775.844677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075776.548703:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075776.548717:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075776.548723:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075776.548737:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2465 00000100:00080000:3.0:1769075776.548751:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075776.548760:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075776.548766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075776.548809:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923654528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075776.548840:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075776.548844:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075776.548846:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075776.548852:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2465 00000100:00080000:3.0:1769075776.548857:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075776.548861:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075776.548871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923654528:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075776.548873:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075776.548890:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923654656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075776.548924:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075776.548927:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075776.548930:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075776.548934:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2465 00000100:00080000:3.0:1769075776.548939:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769075776.548946:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075776.548946:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075776.548953:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075776.548969:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923654784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075776.548989:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075776.548992:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075776/real 1769075776] req@ffff969e0a7fca80 x1855007923654528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075776.548996:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075776.548999:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075776.549009:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2465 00000100:00080000:3.0:1769075776.549013:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075776.549017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923654528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075776.549021:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075776.549032:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075776.549035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075776.549041:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075776.549052:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923654912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075776.549056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923654528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075776.549083:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075776.549087:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075776.549089:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075776.549094:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2465 00000100:00100000:2.0:1769075776.549100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654656:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075776.549109:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075776.549117:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075776.549123:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075776.549128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075776.549143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075776/real 1769075776] req@ffff969e0a7fc000 x1855007923654656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075776.549145:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923655040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075776.549167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923654656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075776.549177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075776.549181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075776.549183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075776.549189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075776.549202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075776.549210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075776/real 1769075776] req@ffff969e0a7fd500 x1855007923654784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075776.549223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923654784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075776.549231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075776.549234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075776.549238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075776.549245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075776.549266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075776.549288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075776/real 1769075776] req@ffff969e0a7fdc00 x1855007923654912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075776.549302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923654912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075776.549311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075776.549313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075776.549315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923654912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075776.549328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075776.549343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075776.549351:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075776/real 1769075776] req@ffff969e0a7ff800 x1855007923655040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075776.549359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923655040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075776.549365:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075776.549367:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075776.549369:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075776.868676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075777.572854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075777.892751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075778.596893:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075778.916695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075779.620816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075779.940726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075780.644813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075780.964650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075781.668681:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075781.668698:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075781.668704:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075781.668719:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2470 00000100:00080000:1.0:1769075781.668735:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075781.668745:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075781.668793:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923655168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075781.668861:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075781.668864:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075781.668866:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075781.668871:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2470 00000100:00080000:1.0:1769075781.668875:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075781.668879:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075781.668890:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075781.668903:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923655296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075781.668911:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075781.668915:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075781.668916:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075781.668919:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2470 00000100:00080000:1.0:1769075781.668922:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075781.668928:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075781.668929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923655168:192.168.204.143@tcp:250:kworker.0 00000008:00100000:1.0:1769075781.668932:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075781.668946:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923655424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075781.668955:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075781.668958:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075781.668959:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075781.668962:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2470 00000100:00080000:1.0:1769075781.668972:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075781.668976:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075781.668981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075781.669010:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923655552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075781.669036:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075781.669039:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075781.669042:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075781.669046:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2470 00000100:00080000:1.0:1769075781.669051:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075781.669054:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075781.669060:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075781.669078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075781.669104:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923655680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075781.669165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075781/real 1769075781] req@ffff969e10b60380 x1855007923655168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075781.669200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923655168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075781.669240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075781.669246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075781.669251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923655168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075781.669260:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075781.669305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075781.669318:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075781/real 1769075781] req@ffff969e10b61180 x1855007923655296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769075781.669321:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075781.669333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923655296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075781.669341:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075781.669345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075781.669350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075781.669374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075781.669388:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075781.669395:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075781/real 1769075781] req@ffff969e10b60000 x1855007923655424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075781.669405:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923655424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075781.669423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075781.669426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075781.669429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075781.669434:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075781.669451:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075781.669458:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075781/real 1769075781] req@ffff969e10b62d80 x1855007923655552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075781.669467:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923655552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075781.669474:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075781.669476:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075781.669478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075781.669482:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075781.669494:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075781.669501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075781/real 1769075781] req@ffff969e10b60700 x1855007923655680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075781.669510:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923655680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075781.669516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075781.669518:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075781.669526:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075781.988766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075782.692871:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075783.012755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075783.717104:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075784.036686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075784.740769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075785.060751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075785.764865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075786.084696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075786.788695:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075786.788711:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075786.788718:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075786.788733:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2475 00000100:00080000:0.0:1769075786.788747:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075786.788755:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075786.788795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075786.788803:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923655808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075786.788830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075786.788854:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075786.788857:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075786.788861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923655808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075786.788865:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2475 00000100:00080000:0.0:1769075786.788872:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075786.788876:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075786.788886:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075786.788915:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923655936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075786.788930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075786.788934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075786.788936:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075786.788956:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2475 00000100:00080000:0.0:1769075786.788969:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075786.788973:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075786.788980:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075786.788994:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075786.788997:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923656064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075786.789007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075786.789011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075786.789013:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075786.789016:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2475 00000100:00080000:0.0:1769075786.789020:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075786.789023:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075786.789035:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075786.789039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075786/real 1769075786] req@ffff969e06a62d80 x1855007923655808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075786.789057:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923656192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075786.789066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075786.789069:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923655808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075786.789070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075786.789074:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075786.789081:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2475 00000100:00080000:0.0:1769075786.789087:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075786.789089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075786.789093:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075786.789098:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075786.789098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075786.789104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923655808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075786.789110:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923656320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075786.789134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075786.789154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075786.789165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075786/real 1769075786] req@ffff969e06a62680 x1855007923655936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075786.789174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923655936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075786.789181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075786.789184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075786.789186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923655936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075786.789189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075786.789214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075786.789221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075786/real 1769075786] req@ffff969e06a61f80 x1855007923656064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075786.789229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923656064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075786.789234:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075786.789236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075786.789238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075786.789242:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075786.789254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075786.789260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075786/real 1769075786] req@ffff969e06a60a80 x1855007923656192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075786.789267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923656192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075786.789273:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075786.789275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075786.789277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075786.789280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075786.789290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075786.789295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075786/real 1769075786] req@ffff969e06a63800 x1855007923656320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075786.789313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923656320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075786.789320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075786.789322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075786.789324:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075787.108733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075787.812846:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075788.132675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075788.836794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075789.156764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075789.860890:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075790.180675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075790.884913:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075791.204741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075791.908762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075791.908773:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075791.908778:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075791.908788:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2481 00000100:00080000:0.0:1769075791.908808:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075791.908828:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075791.908890:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923656448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075791.908939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075791.908964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:2.0:1769075791.908967:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075791.908970:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075791.908979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923656448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075791.908989:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2481 00000100:00080000:0.0:1769075791.909019:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075791.909027:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075791.909037:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075791.909061:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923656576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075791.909074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075791.909077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075791.909079:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075791.909082:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2481 00000100:00080000:0.0:1769075791.909086:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075791.909090:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075791.909095:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075791.909106:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923656704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075791.909116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075791.909119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075791.909121:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075791.909124:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2481 00000100:00080000:0.0:1769075791.909127:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075791.909129:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075791.909136:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075791.909149:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075791.909166:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923656832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075791.909177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075791.909180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075791.909182:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075791.909188:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2481 00000100:00080000:0.0:1769075791.909195:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075791.909200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075791/real 1769075791] req@ffff969e06a60000 x1855007923656448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075791.909205:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075791.909216:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075791.909238:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923656960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075791.909251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923656448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075791.909266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075791.909271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075791.909281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923656448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075791.909318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075791.909356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075791.909376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075791/real 1769075791] req@ffff969e06a60700 x1855007923656576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075791.909395:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923656576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075791.909417:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075791.909421:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075791.909424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075791.909431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075791.909453:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075791.909463:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075791/real 1769075791] req@ffff969e06a63800 x1855007923656704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075791.909492:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923656704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075791.909501:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075791.909504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075791.909507:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075791.909513:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075791.909534:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075791.909544:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075791/real 1769075791] req@ffff969e06a60a80 x1855007923656832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075791.909554:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923656832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075791.909563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075791.909565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075791.909568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075791.909574:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075791.909589:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075791.909645:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075791/real 1769075791] req@ffff969e06a61f80 x1855007923656960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075791.909665:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923656960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075791.909704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075791.909708:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075791.909711:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923656960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075792.228758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075792.932874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075793.252731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075793.956884:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075794.276872:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075794.980823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075795.300831:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075796.004877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075796.324673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075797.028720:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075797.028749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075797.028754:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075797.028767:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2486 00010000:00010000:0.0:1769075797.028776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075797.028783:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075797.028792:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075797.028844:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923657088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075797.028874:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075797.028878:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075797.028879:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075797.028883:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2486 00000100:00080000:2.0:1769075797.028886:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075797.028891:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075797.028903:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075797.028919:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923657088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075797.028947:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923657216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075797.028957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075797.028960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075797.028962:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075797.028966:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2486 00000100:00080000:2.0:1769075797.028969:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075797.028973:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075797.028978:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075797.028988:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923657344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075797.028996:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075797.028999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075797.029001:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075797.029003:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2486 00000100:00080000:2.0:1769075797.029006:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075797.029012:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075797.029012:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075797.029020:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075797.029029:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923657472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075797.029037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075797.029040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075797.029042:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075797.029053:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2486 00000100:00080000:2.0:1769075797.029058:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075797.029059:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075797/real 1769075797] req@ffff969e0a46f100 x1855007923657088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075797.029066:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075797.029077:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075797.029086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923657088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075797.029091:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923657600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075797.029107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075797.029116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075797.029120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923657088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075797.029149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075797.029167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075797.029176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075797/real 1769075797] req@ffff969e0a46f800 x1855007923657216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075797.029186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923657216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075797.029201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075797.029204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075797.029207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075797.029212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075797.029228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075797.029236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075797/real 1769075797] req@ffff969e0a46c380 x1855007923657344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075797.029245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923657344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075797.029251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075797.029253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075797.029255:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075797.029260:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075797.029273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075797.029280:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075797/real 1769075797] req@ffff969e0a46d500 x1855007923657472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075797.029288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923657472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075797.029295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075797.029297:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075797.029305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075797.029310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075797.029326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075797.029333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075797/real 1769075797] req@ffff969e0a46f480 x1855007923657600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075797.029342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923657600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075797.029349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075797.029351:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075797.029353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075797.348690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075798.052878:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075798.372782:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075799.076990:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075799.396705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075800.100837:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075800.420895:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075801.124821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075801.444809:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075802.148806:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075802.148824:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075802.148832:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075802.148851:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2491 00010000:00010000:0.0:1769075802.148858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075802.148871:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075802.148884:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075802.148950:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923657728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075802.148997:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075802.149003:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075802.149006:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075802.149014:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2491 00000100:00080000:3.0:1769075802.149021:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075802.149030:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075802.149047:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075802.149103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923657728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075802.149104:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923657856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075802.149136:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075802.149166:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075802.149169:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075802.149178:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2491 00000100:00080000:3.0:1769075802.149186:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075802.149193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075802.149200:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075802.149211:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075802.149233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075802/real 1769075802] req@ffff969e0a7ffb80 x1855007923657728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075802.149240:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923657984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075802.149274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923657728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075802.149274:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075802.149288:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075802.149293:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075802.149310:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2491 00000100:00080000:1.0:1769075802.149315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075802.149325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769075802.149332:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075802.149345:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075802.149353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923657728:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075802.149361:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075802.149379:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075802.149386:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923658112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075802.149420:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075802.149431:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075802.149437:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075802/real 1769075802] req@ffff969e0a7ff800 x1855007923657856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075802.149441:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075802.149450:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075802.149462:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923657856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075802.149467:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2491 00000100:00080000:1.0:1769075802.149486:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075802.149486:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075802.149513:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769075802.149516:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075802.149521:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075802.149531:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657984:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769075802.149543:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075802.149558:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075802.149578:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075802/real 1769075802] req@ffff969e0a7fdc00 x1855007923657984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075802.149580:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923658240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075802.149598:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923657984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075802.149672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075802.149705:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075802.149720:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923657984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075802.149767:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075802.149790:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075802.149810:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075802/real 1769075802] req@ffff969e0a7fd500 x1855007923658112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075802.149821:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923658112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075802.149828:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075802.149832:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075802.149834:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075802.149839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075802.149854:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075802.149862:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075802/real 1769075802] req@ffff969e0a7fc000 x1855007923658240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075802.149871:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923658240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075802.149878:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075802.149881:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075802.149883:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075802.468814:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075803.172801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075803.492780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075804.196769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075804.516702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075805.220905:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075805.540707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075806.244781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075806.564691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075807.268737:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075807.268753:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075807.268788:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075807.268813:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2496 00000100:00080000:1.0:1769075807.268830:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075807.268840:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075807.268903:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923658368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075807.268944:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075807.268949:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075807.268952:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075807.268958:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2496 00000100:00080000:1.0:1769075807.268962:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075807.268967:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075807.268980:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075807.268997:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923658496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075807.269012:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075807.269016:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075807.269018:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075807.269024:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2496 00000100:00080000:1.0:1769075807.269027:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075807.269032:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075807.269039:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075807.269046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923658368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075807.269057:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923658624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075807.269093:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075807.269098:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075807.269101:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075807.269107:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2496 00000100:00080000:1.0:1769075807.269114:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075807.269121:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075807.269131:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769075807.269159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075807.269163:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923658752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075807.269188:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075807.269196:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075807.269201:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075807.269211:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2496 00000100:00080000:1.0:1769075807.269221:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075807.269223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075807/real 1769075807] req@ffff969e10b62680 x1855007923658368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075807.269238:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075807.269254:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075807.269263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923658368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075807.269281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769075807.269285:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923658880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075807.269292:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075807.269307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923658368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075807.269329:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075807.269364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075807.269415:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075807/real 1769075807] req@ffff969e10b62a00 x1855007923658496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075807.269446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923658496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075807.269467:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075807.269494:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00010000:00010000:0.0:1769075807.269496:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075807.269508:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075807.269644:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075807.269682:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075807.269699:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075807/real 1769075807] req@ffff969e10b60a80 x1855007923658624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075807.269716:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923658624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075807.269727:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075807.269731:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075807.269735:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075807.269743:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075807.269764:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075807.269775:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075807/real 1769075807] req@ffff969e10b61f80 x1855007923658752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075807.269789:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923658752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075807.269800:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075807.269804:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075807.269807:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075807.269815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075807.269868:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075807.269883:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075807/real 1769075807] req@ffff969e10b61500 x1855007923658880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075807.269898:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923658880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075807.269911:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075807.269918:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075807.269925:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923658880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075807.588710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075808.292791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075808.612770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075809.316976:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075809.636753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075810.340732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075810.660691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075811.364737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075811.684752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075812.388700:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075812.388710:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769075812.388714:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075812.388723:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2501 00000100:00080000:1.0:1769075812.388734:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075812.388740:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075812.388779:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923659008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075812.388829:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075812.388835:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075812.388838:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075812.388844:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2501 00000100:00080000:1.0:1769075812.388865:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075812.388872:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075812.388886:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075812.388910:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923659136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075812.388918:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923659008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769075812.388949:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075812.388956:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075812.388960:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075812.388969:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2501 00000100:00080000:1.0:1769075812.388975:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075812.388982:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075812.389007:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075812.389040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075812.389046:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923659264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075812.389068:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075812.389074:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075812.389078:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075812.389084:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2501 00000100:00080000:1.0:1769075812.389091:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075812.389097:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075812.389106:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075812.389106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075812/real 1769075812] req@ffff969e10b61500 x1855007923659008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075812.389129:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923659392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075812.389145:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075812.389150:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769075812.389152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923659008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075812.389155:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075812.389171:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2501 00000100:00080000:2.0:1769075812.389174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075812.389178:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075812.389183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:1.0:1769075812.389192:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075812.389202:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075812.389203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923659008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075812.389237:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923659520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075812.389252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075812.389301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075812.389322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075812/real 1769075812] req@ffff969e10b61f80 x1855007923659136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075812.389358:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923659136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075812.389373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075812.389378:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075812.389384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659136:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769075812.389387:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075812.389395:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075812.389431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075812.389442:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075812/real 1769075812] req@ffff969e10b60a80 x1855007923659264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075812.389455:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923659264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075812.389463:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075812.389466:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075812.389509:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075812.389515:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075812.389534:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075812.389544:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075812/real 1769075812] req@ffff969e10b62a00 x1855007923659392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075812.389554:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923659392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075812.389602:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075812.389606:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075812.389610:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075812.389677:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075812.389698:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075812.389707:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075812/real 1769075812] req@ffff969e10b62680 x1855007923659520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075812.389717:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923659520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075812.389725:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075812.389727:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075812.389730:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075812.708677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075813.412849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075813.732683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075814.436877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075814.756759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075815.460813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075815.780713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075816.484806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075816.804666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075817.508777:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075817.508800:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075817.508805:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075817.508816:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2506 00000100:00080000:2.0:1769075817.508828:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075817.508836:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075817.508890:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769075817.508917:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923659648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075817.508968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075817.509004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769075817.509012:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923659648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075817.509012:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075817.509030:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2506 00000100:00080000:2.0:1769075817.509037:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075817.509044:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075817.509057:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075817.509094:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923659776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075817.509107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075817.509110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075817.509112:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075817.509116:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2506 00000100:00080000:2.0:1769075817.509121:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075817.509127:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075817.509138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769075817.509142:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075817.509158:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923659904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075817.509169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075817.509173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075817.509176:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075817.509182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075817/real 1769075817] req@ffff969e0a46df80 x1855007923659648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075817.509188:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2506 00000100:00080000:2.0:1769075817.509203:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075817.509211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923659648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075817.509214:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769075817.509224:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075817.509234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769075817.509238:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075817.509240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923659648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075817.509283:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923660032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075817.509296:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075817.509299:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075817.509302:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075817.509303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659776:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769075817.509319:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2506 00000100:00080000:2.0:1769075817.509326:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075817.509331:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075817.509337:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075817.509344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075817.509364:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923660160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075817.509366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075817/real 1769075817] req@ffff969e0a46ea00 x1855007923659776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075817.509568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923659776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075817.509579:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075817.509583:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075817.509587:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075817.509622:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075817.509648:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075817.509674:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075817/real 1769075817] req@ffff969e0a46ca80 x1855007923659904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075817.509698:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923659904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075817.509704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075817.509706:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075817.509708:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923659904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075817.509713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075817.509751:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075817.509762:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075817/real 1769075817] req@ffff969e0a46d880 x1855007923660032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075817.509774:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923660032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075817.509782:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075817.509785:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075817.509788:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075817.509807:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075817.509823:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075817.509833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075817/real 1769075817] req@ffff969e0a46ed80 x1855007923660160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075817.509844:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923660160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075817.509877:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075817.509880:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075817.509882:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075817.828722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075818.532860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075818.852750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075819.556886:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075819.876742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075820.580751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075820.900775:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075821.604864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075821.924727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075822.628812:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075822.628831:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075822.628839:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075822.628854:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2511 00000100:00080000:3.0:1769075822.628874:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075822.628897:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075822.628899:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075822.628967:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923660288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075822.629031:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075822.629036:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075822.629039:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075822.629045:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2511 00000100:00080000:3.0:1769075822.629049:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075822.629054:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075822.629069:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075822.629087:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923660416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075822.629101:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075822.629105:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075822.629107:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075822.629112:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2511 00000100:00080000:3.0:1769075822.629155:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075822.629158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923660288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075822.629165:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075822.629179:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075822.629199:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923660544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075822.629215:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075822.629219:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075822.629222:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075822.629227:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2511 00000100:00080000:3.0:1769075822.629232:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075822.629236:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075822.629242:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075822.629257:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923660672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075822.629267:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075822.629271:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075822.629274:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075822.629278:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2511 00000100:00080000:3.0:1769075822.629286:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075822.629294:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075822.629350:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075822.629377:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075822.629412:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923660800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075822.629447:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075822/real 1769075822] req@ffff969e0a7fca80 x1855007923660288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075822.629484:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923660288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075822.629500:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075822.629507:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075822.629513:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923660288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075822.629525:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075822.629559:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075822.629575:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075822/real 1769075822] req@ffff969e0a7fce00 x1855007923660416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075822.629625:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923660416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075822.629651:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075822.629656:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075822.629661:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075822.629699:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075822.629730:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075822.629745:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075822/real 1769075822] req@ffff969e0a7fd180 x1855007923660544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075822.629760:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923660544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075822.629771:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075822.629775:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075822.629780:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075822.629786:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075822.629806:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075822.629816:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075822/real 1769075822] req@ffff969e0a7fdf80 x1855007923660672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075822.629831:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923660672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075822.629846:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075822.629852:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075822.629874:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075822.629887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075822.629932:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075822.629954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075822/real 1769075822] req@ffff969e0a7fc380 x1855007923660800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075822.629978:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923660800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075822.629990:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075822.629994:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075822.629999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923660800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075822.948672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075823.652863:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075823.972736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075824.676975:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075824.996673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075825.700824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075826.020694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075826.724790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075827.044681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075827.748742:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075827.748753:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075827.748757:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075827.748768:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2516 00000100:00080000:3.0:1769075827.748779:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075827.748787:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075827.748826:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923660928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075827.748840:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075827.748855:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075827.748860:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075827.748862:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075827.748865:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2516 00000100:00080000:3.0:1769075827.748868:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075827.748873:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075827.748883:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075827.748893:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923661056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075827.748903:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075827.748906:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075827.748908:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075827.748915:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2516 00000100:00080000:3.0:1769075827.748918:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075827.748928:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075827.748929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923660928:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075827.748951:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075827.748968:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923661184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075827.748977:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075827.748980:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075827.748981:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075827.748984:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2516 00000100:00080000:3.0:1769075827.748988:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075827.748992:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075827.748996:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075827.749006:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923661312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075827.749014:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075827.749016:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075827.749018:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075827.749020:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2516 00000100:00080000:3.0:1769075827.749024:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075827.749027:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075827.749032:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075827.749044:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923661440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075827.749058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075827.749137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075827/real 1769075827] req@ffff969e0a7fe300 x1855007923660928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075827.749181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923660928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075827.749198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075827.749205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075827.749270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923660928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075827.749303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075827.749330:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075827.749344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075827/real 1769075827] req@ffff969e0a7fea00 x1855007923661056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075827.749356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923661056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075827.749364:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075827.749368:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075827.749371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075827.749377:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075827.749392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075827.749400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075827/real 1769075827] req@ffff969e0a7fd880 x1855007923661184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075827.749410:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923661184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075827.749417:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075827.749420:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075827.749423:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075827.749428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075827.749442:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075827.749460:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075827/real 1769075827] req@ffff969e0a7fe680 x1855007923661312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075827.749472:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923661312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075827.749480:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075827.749483:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075827.749485:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075827.749491:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075827.749507:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075827.749515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075827/real 1769075827] req@ffff969e0a7fc700 x1855007923661440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075827.749525:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923661440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075827.749533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075827.749535:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075827.749538:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075828.068820:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075828.772829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075829.092699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075829.796781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075830.116692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075830.820811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075831.140675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075831.844908:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075832.164736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075832.868698:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075832.868709:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075832.868733:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075832.868742:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2522 00000100:00080000:3.0:1769075832.868753:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075832.868761:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075832.868792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075832.868799:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923661568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075832.868828:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075832.868832:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075832.868833:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075832.868837:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2522 00000100:00080000:3.0:1769075832.868840:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075832.868844:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075832.868854:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075832.868868:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923661696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075832.868876:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923661568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075832.868884:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075832.868889:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075832.868892:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075832.868899:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2522 00000100:00080000:3.0:1769075832.868905:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075832.868910:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075832.868918:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075832.868936:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923661824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075832.868972:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075832.868978:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075832.868981:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075832.868984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661696:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075832.869000:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2522 00000100:00080000:3.0:1769075832.869011:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075832.869018:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075832.869025:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075832.869043:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923661952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075832.869062:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075832.869063:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661824:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075832.869074:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075832.869078:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075832.869083:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2522 00000100:00080000:3.0:1769075832.869088:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075832.869092:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075832.869098:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075832.869107:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769075832.869114:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923662080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075832.869189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075833.188742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075833.892915:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075834.212735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075834.916934:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075835.236763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075835.941108:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769075835.941173:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:1.0:1769075835.941254:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769075835.941284:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769075835.941345:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769075835.941368:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769075835.941378:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769075835.941390:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769075835.941462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075832/real 1769075835] req@ffff969e0a7fea00 x1855007923662080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075835.941515:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923662080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075835.941540:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075835.941550:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075835.941560:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075835.941575:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075832/real 1769075835] req@ffff969e0a7fd880 x1855007923661952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075835.941652:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923661952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075835.941709:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075835.941718:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075835.941726:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075835.941740:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075832/real 1769075835] req@ffff969e0a7fc700 x1855007923661696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075835.941766:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923661696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075835.941781:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075835.941787:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075835.941794:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075835.941807:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075832/real 1769075835] req@ffff969e0a7fe680 x1855007923661824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075835.941829:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923661824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075835.941845:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075835.941851:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075835.941856:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923661824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075835.941868:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075832/real 1769075835] req@ffff969e0a7fed80 x1855007923661568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075835.941888:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923661568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075835.941938:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075835.941945:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075835.941950:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923661568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075836.260699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075836.964799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075837.284745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075837.988686:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075837.988718:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075837.988724:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075837.988738:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2527 00000100:00080000:2.0:1769075837.988755:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075837.988781:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075837.988854:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923662208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769075837.988857:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075837.988917:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075837.988922:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075837.988924:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075837.988929:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2527 00000100:00080000:2.0:1769075837.988934:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075837.988939:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075837.988953:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075837.988977:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923662336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075837.988978:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923662208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075837.989004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075837.989010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075837.989066:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075837.989073:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2527 00000100:00080000:2.0:1769075837.989078:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075837.989084:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075837.989091:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075837.989108:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923662464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075837.989121:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075837.989129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075837.989130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075837.989135:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075837.989141:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2527 00000100:00080000:2.0:1769075837.989150:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075837.989157:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075837.989166:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075837.989181:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923662592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075837.989192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075837/real 1769075837] req@ffff969e0a46e300 x1855007923662208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075837.989193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075837.989203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075837.989208:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075837.989217:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2527 00000100:00080000:2.0:1769075837.989222:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075837.989227:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075837.989232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923662208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769075837.989237:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075837.989247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075837.989255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075837.989264:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923662720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075837.989274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923662208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075837.989318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075837.989350:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075837.989365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075837/real 1769075837] req@ffff969e0a46ce00 x1855007923662336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075837.989380:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923662336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075837.989389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075837.989393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075837.989397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075837.989404:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075837.989425:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075837.989436:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075837/real 1769075837] req@ffff969e0a46ed80 x1855007923662464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075837.989450:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923662464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075837.989459:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075837.989463:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075837.989467:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075837.989474:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075837.989495:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075837.989516:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075837/real 1769075837] req@ffff969e0a46d880 x1855007923662592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075837.989531:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923662592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075837.989541:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075837.989545:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075837.989548:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075837.989555:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075837.989578:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075837.989617:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075837/real 1769075837] req@ffff969e0a46ca80 x1855007923662720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075837.989632:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923662720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075837.989643:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075837.989647:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075837.989650:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075838.308701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075839.013090:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075839.332699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075840.036875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075840.356678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075841.060875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075841.380703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075842.084862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075842.404826:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075843.108741:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1769075843.108750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075843.108751:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075843.108770:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075843.108796:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2532 00000100:00080000:3.0:1769075843.108818:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075843.108825:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075843.108871:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923662848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075843.108897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075843.108900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075843.108902:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075843.108905:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2532 00000100:00080000:3.0:1769075843.108908:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075843.108912:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075843.108920:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075843.108930:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923662976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075843.108940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075843.108946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769075843.108947:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923662848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075843.108948:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075843.108978:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2532 00000100:00080000:3.0:1769075843.109000:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075843.109003:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075843.109008:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075843.109020:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923663104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075843.109038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075843.109047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075843.109048:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075843.109051:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2532 00000100:00080000:3.0:1769075843.109056:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075843.109059:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075843.109063:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075843.109074:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075843.109076:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923663232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075843.109084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075843.109087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075843.109088:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075843.109092:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2532 00000100:00080000:3.0:1769075843.109096:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075843.109099:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075843.109102:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075843.109112:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923663360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075843.109130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075843/real 1769075843] req@ffff969e0a7fe300 x1855007923662848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075843.109148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923662848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075843.109158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075843.109162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075843.109166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923662848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075843.109188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075843.109210:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075843.109219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075843/real 1769075843] req@ffff969e0a7fc380 x1855007923662976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075843.109229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923662976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075843.109236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075843.109238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075843.109240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923662976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075843.109245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075843.109277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075843.109284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075843/real 1769075843] req@ffff969e0a7fdf80 x1855007923663104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075843.109293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923663104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075843.109310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075843.109313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075843.109327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075843.109331:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075843.109356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075843.109380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075843/real 1769075843] req@ffff969e0a7fd180 x1855007923663232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075843.109399:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923663232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075843.109406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075843.109408:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075843.109411:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075843.109415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075843.109427:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075843.109434:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075843/real 1769075843] req@ffff969e0a7fce00 x1855007923663360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075843.109442:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923663360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075843.109459:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075843.109461:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075843.109469:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075843.428734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075844.132823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075844.452738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075845.156839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075845.476707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075846.180840:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075846.500822:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075847.204804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075847.524702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075848.228695:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075848.228708:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075848.228713:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075848.228724:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2537 00000100:00080000:3.0:1769075848.228737:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075848.228745:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075848.228784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075848.228793:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923663488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075848.228842:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075848.228845:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075848.228847:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075848.228852:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2537 00000100:00080000:3.0:1769075848.228857:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075848.228864:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075848.228891:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075848.228916:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923663616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075848.228917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923663488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075848.228944:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075848.228947:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075848.228950:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075848.228955:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2537 00000100:00080000:3.0:1769075848.228959:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075848.228963:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075848.228968:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075848.228980:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923663744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075848.228987:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075848.228990:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075848.228991:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075848.228995:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2537 00000100:00080000:3.0:1769075848.229006:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075848.229015:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075848.229017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769075848.229022:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075848.229036:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923663872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075848.229045:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075848.229049:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075848.229051:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075848.229056:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2537 00000100:00080000:3.0:1769075848.229059:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075848.229063:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075848.229067:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075848.229069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075848/real 1769075848] req@ffff969e0a7fca80 x1855007923663488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075848.229079:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923664000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075848.229137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923663488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075848.229150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075848.229155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075848.229160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923663488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075848.229196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075848.229221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075848.229246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075848/real 1769075848] req@ffff969e0a7fc000 x1855007923663616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075848.229258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923663616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075848.229266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075848.229270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075848.229289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075848.229296:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075848.229318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075848.229329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075848/real 1769075848] req@ffff969e0a7fd500 x1855007923663744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075848.229341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923663744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075848.229349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075848.229352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075848.229356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075848.229362:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075848.229380:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075848.229391:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075848/real 1769075848] req@ffff969e0a7fdc00 x1855007923663872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075848.229403:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923663872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075848.229411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075848.229415:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075848.229418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923663872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075848.229425:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075848.229444:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075848.229462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075848/real 1769075848] req@ffff969e0a7ff800 x1855007923664000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075848.229474:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923664000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075848.229483:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075848.229486:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075848.229490:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075848.548730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075849.252812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075849.572713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075850.276821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075850.596682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075851.300737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075851.620659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075852.324880:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075852.644701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075853.348754:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075853.348766:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075853.348770:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075853.348836:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2542 00000100:00080000:3.0:1769075853.348846:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075853.348853:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075853.348899:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923664128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075853.348922:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075853.348957:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075853.348959:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075853.348965:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2542 00000100:00080000:3.0:1769075853.348969:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075853.348976:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075853.348987:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075853.348994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923664128:192.168.204.143@tcp:250:kworker.0 00010000:00010000:0.0:1769075853.349005:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075853.349009:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923664256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075853.349029:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075853.349033:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075853.349036:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075853.349042:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2542 00000100:00080000:3.0:1769075853.349047:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075853.349053:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075853.349072:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075853.349092:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923664384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075853.349106:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075853.349127:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075853.349129:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075853.349133:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2542 00000100:00080000:3.0:1769075853.349143:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075853.349147:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075853.349159:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075853.349167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075853.349176:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923664512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075853.349186:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075853.349189:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075853.349190:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075853.349195:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2542 00000100:00080000:3.0:1769075853.349207:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075853.349212:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075853.349218:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075853.349238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075853/real 1769075853] req@ffff969e0a7ffb80 x1855007923664128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075853.349240:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923664640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075853.349360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923664128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075853.349382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075853.349391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075853.349399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923664128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075853.349447:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075853.349484:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075853.349523:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075853/real 1769075853] req@ffff969e0a7ff800 x1855007923664256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075853.349540:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923664256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075853.349551:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075853.349556:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075853.349560:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075853.349568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075853.349648:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075853.349664:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075853/real 1769075853] req@ffff969e0a7fdc00 x1855007923664384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075853.349681:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923664384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075853.349696:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075853.349703:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075853.349709:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075853.349721:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075853.349765:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075853.349832:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075853/real 1769075853] req@ffff969e0a7fd500 x1855007923664512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075853.349857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923664512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075853.349878:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075853.349883:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075853.349887:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075853.349895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075853.349927:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075853.349940:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075853/real 1769075853] req@ffff969e0a7fc000 x1855007923664640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075853.349971:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923664640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075853.349983:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075853.349987:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075853.349991:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075853.668698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075854.372828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075854.692705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075855.396816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075855.716739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075856.420770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075856.740658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075857.444835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075857.764686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075858.468752:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075858.468764:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075858.468769:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075858.468779:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2547 00000100:00080000:3.0:1769075858.468793:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075858.468801:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769075858.468833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075858.468865:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923664768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075858.468902:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075858.468906:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075858.468910:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075858.468915:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2547 00000100:00080000:3.0:1769075858.468938:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075858.468943:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075858.468953:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075858.468993:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923664896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075858.469008:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075858.469012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075858.469016:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075858.469021:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2547 00000100:00080000:3.0:1769075858.469029:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075858.469036:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075858.469045:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075858.469060:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923665024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075858.469071:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075858.469078:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075858.469082:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075858.469092:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2547 00000100:00080000:3.0:1769075858.469102:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075858.469110:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075858.469119:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075858.469140:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923665152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075858.469168:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075858.469178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923664768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075858.469182:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075858.469189:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075858.469199:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2547 00000100:00080000:3.0:1769075858.469203:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075858.469207:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075858.469214:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075858.469238:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923665280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075858.469303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075858.469354:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075858/real 1769075858] req@ffff969e0a7fca80 x1855007923664768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075858.469375:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923664768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075858.469424:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075858.469430:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075858.469435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923664768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075858.469444:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075858.469471:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075858.469484:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075858/real 1769075858] req@ffff969e0a7fce00 x1855007923664896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075858.469497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923664896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075858.469506:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075858.469509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075858.469512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923664896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075858.469518:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075858.469534:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075858.469543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075858/real 1769075858] req@ffff969e0a7fd180 x1855007923665024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075858.469555:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923665024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075858.469563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075858.469566:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075858.469569:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075858.469685:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075858.469756:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075858.469779:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075858/real 1769075858] req@ffff969e0a7fdf80 x1855007923665152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075858.469792:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923665152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075858.469802:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075858.469805:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075858.469809:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075858.469815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075858.469855:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075858.469867:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075858/real 1769075858] req@ffff969e0a7fc380 x1855007923665280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075858.469881:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923665280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075858.469909:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075858.469913:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075858.469916:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075858.788770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075859.492821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075859.812660:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075860.516915:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075860.836794:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075861.540883:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075861.860714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075862.564858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075862.884759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769075863.588903:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075863.588920:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1769075863.588921:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769075863.588928:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075863.588947:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2552 00000100:00080000:1.0:1769075863.588968:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075863.588979:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075863.589045:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923665408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075863.589089:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075863.589096:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769075863.589100:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075863.589108:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2552 00000100:00080000:1.0:1769075863.589115:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075863.589125:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075863.589146:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075863.589171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923665408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075863.589181:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923665536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075863.589209:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075863.589217:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769075863.589221:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075863.589231:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2552 00000100:00080000:1.0:1769075863.589240:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075863.589248:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075863.589289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:1.0:1769075863.589296:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075863.589328:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923665664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075863.589344:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075863.589350:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769075863.589353:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075863.589359:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2552 00000100:00080000:1.0:1769075863.589368:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075863.589370:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075863/real 1769075863] req@ffff969e10b60700 x1855007923665408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075863.589378:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075863.589393:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075863.589414:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923665792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075863.589428:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769075863.589438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923665408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075863.589439:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075863.589446:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075863.589459:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2552 00000100:00080000:1.0:1769075863.589468:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075863.589480:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769075863.589488:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075863.589489:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075863.589504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075863.589522:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923665408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075863.589525:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923665920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075863.589587:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075863.589704:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075863.589724:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075863/real 1769075863] req@ffff969e10b62d80 x1855007923665536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075863.589742:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923665536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075863.589754:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075863.589759:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075863.589764:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075863.589773:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075863.589822:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075863.589842:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075863/real 1769075863] req@ffff969e10b60000 x1855007923665664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075863.589868:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923665664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075863.589884:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075863.589891:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075863.589898:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075863.589911:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075863.589953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075863.589974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075863/real 1769075863] req@ffff969e10b61180 x1855007923665792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075863.589998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923665792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075863.590011:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075863.590015:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075863.590019:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075863.590027:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075863.590057:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075863.590069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075863/real 1769075863] req@ffff969e10b60380 x1855007923665920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075863.590105:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923665920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075863.590117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075863.590121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075863.590125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923665920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075863.908692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075864.612802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075864.932782:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075865.636940:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075865.956721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075866.660778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075866.980657:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075867.684785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075868.004677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075868.708716:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075868.708725:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075868.708729:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075868.708737:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2557 00000100:00080000:0.0:1769075868.708747:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075868.708754:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075868.708767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075868.708794:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923666048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075868.708840:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075868.708843:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075868.708844:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075868.708849:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2557 00000100:00080000:0.0:1769075868.708853:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075868.708857:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075868.708866:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075868.708868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923666048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075868.708880:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923666176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075868.708889:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075868.708892:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075868.708894:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075868.708899:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2557 00000100:00080000:0.0:1769075868.708903:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075868.708908:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075868.708913:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769075868.708944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075868.708945:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923666304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075868.708960:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075868.708964:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075868.708966:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075868.708971:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2557 00000100:00100000:2.0:1769075868.708975:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075868/real 1769075868] req@ffff969e06a62680 x1855007923666048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075868.709000:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075868.709006:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075868.709011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923666048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075868.709012:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769075868.709024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075868.709030:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769075868.709032:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923666432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075868.709035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923666048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075868.709051:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075868.709055:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075868.709058:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769075868.709059:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666176:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075868.709066:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2557 00000100:00080000:0.0:1769075868.709073:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075868.709078:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769075868.709083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769075868.709086:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075868.709095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075868/real 1769075868] req@ffff969e06a62d80 x1855007923666176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075868.709102:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923666560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075868.709139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923666176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075868.709146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075868.709148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075868.709151:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075868.709155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075868.709168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075868.709175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075868/real 1769075868] req@ffff969e06a61500 x1855007923666304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075868.709182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923666304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075868.709187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075868.709189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075868.709190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075868.709194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075868.709204:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075868.709209:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075868/real 1769075868] req@ffff969e06a61180 x1855007923666432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075868.709216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923666432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075868.709230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075868.709233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075868.709235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075868.709246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075868.709259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075868.709265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075868/real 1769075868] req@ffff969e06a60380 x1855007923666560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075868.709272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923666560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075868.709277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075868.709279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075868.709281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075869.028678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075869.732829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075870.052666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075870.756812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075871.076696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075871.780903:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075872.100722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075872.804885:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075873.124733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075873.828729:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075873.828743:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075873.828750:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075873.828764:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2563 00000100:00080000:0.0:1769075873.828778:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075873.828788:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075873.828826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075873.828848:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923666688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075873.828927:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075873.828935:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075873.828939:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075873.828948:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2563 00000100:00080000:0.0:1769075873.828956:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075873.828963:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075873.828977:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075873.829004:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923666688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075873.829005:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923666816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075873.829032:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075873.829036:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075873.829039:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075873.829045:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2563 00000100:00080000:0.0:1769075873.829050:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075873.829055:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075873.829063:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075873.829078:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923666944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075873.829092:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075873.829097:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075873.829107:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769075873.829107:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075873.829121:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2563 00000100:00080000:0.0:1769075873.829140:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075873.829148:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075873.829158:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075873.829164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075873/real 1769075873] req@ffff969e06a63100 x1855007923666688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075873.829185:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923667072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769075873.829196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923666688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075873.829238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075873.829242:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075873.829249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075873.829255:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075873.829261:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075873.829277:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2563 00000100:00080000:0.0:1769075873.829285:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769075873.829293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923666688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075873.829302:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075873.829317:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075873.829336:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075873.829349:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923667200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769075873.829380:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075873.829402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075873/real 1769075873] req@ffff969e06a61880 x1855007923666816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075873.829421:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923666816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075873.829432:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075873.829484:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075873.829491:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075873.829500:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769075873.829529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075873.829543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075873/real 1769075873] req@ffff969e06a60380 x1855007923666944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075873.829558:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923666944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075873.829568:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075873.829572:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075873.829576:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923666944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769075873.829583:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075873.829650:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075873.829679:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075873/real 1769075873] req@ffff969e06a61180 x1855007923667072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075873.829695:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923667072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075873.829706:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075873.829710:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075873.829714:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769075873.829736:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769075873.829775:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769075873.829788:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075873/real 1769075873] req@ffff969e06a61500 x1855007923667200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769075873.829803:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923667200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075873.829813:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075873.829817:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075873.829821:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075874.148675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075874.852831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075875.172682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075875.876807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075876.196681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075876.900811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075877.220721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075877.924790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075878.244691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075878.948763:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075878.948773:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075878.948778:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075878.948788:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2568 00000100:00080000:0.0:1769075878.948798:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075878.948805:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075878.948844:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923667328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769075878.948862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075878.948876:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075878.948898:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075878.948900:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075878.948906:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2568 00000100:00080000:0.0:1769075878.948910:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075878.948918:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075878.948920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923667328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769075878.948928:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075878.948945:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923667456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075878.948956:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075878.948959:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075878.948961:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075878.948965:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2568 00000100:00080000:0.0:1769075878.948978:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075878.948981:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075878.948985:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075878.948992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075878.948997:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923667584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075878.949006:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075878.949010:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075878.949012:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075878.949015:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2568 00000100:00080000:0.0:1769075878.949018:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075878.949021:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075878.949032:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075878.949041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075878/real 1769075878] req@ffff969e06a62d80 x1855007923667328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075878.949044:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923667712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075878.949076:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075878.949080:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075878.949082:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075878.949086:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2568 00000100:00080000:0.0:1769075878.949090:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075878.949095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923667328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075878.949098:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075878.949104:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075878.949117:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923667840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075878.949118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075878.949130:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075878.949138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923667328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075878.949162:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075878.949185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075878.949195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075878/real 1769075878] req@ffff969e06a62680 x1855007923667456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075878.949222:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923667456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075878.949229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075878.949232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075878.949235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075878.949240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075878.949256:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075878.949264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075878/real 1769075878] req@ffff969e06a61f80 x1855007923667584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075878.949273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923667584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075878.949279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075878.949282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075878.949284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075878.949288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075878.949300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075878.949307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075878/real 1769075878] req@ffff969e06a60a80 x1855007923667712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075878.949315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923667712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075878.949333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075878.949336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075878.949339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075878.949343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075878.949420:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075878.949432:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075878/real 1769075878] req@ffff969e06a63800 x1855007923667840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075878.949442:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923667840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075878.949450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075878.949453:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075878.949456:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923667840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075879.268762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075879.972909:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075880.292758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075880.996836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075881.316772:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075882.020889:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075882.340722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075883.044829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075883.364739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075884.068757:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075884.068771:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075884.068787:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075884.068801:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2573 00000100:00080000:0.0:1769075884.068830:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075884.068841:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075884.068859:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075884.068899:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923667968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075884.068937:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075884.068973:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075884.068977:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075884.068984:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2573 00000100:00080000:0.0:1769075884.068991:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075884.069003:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075884.069006:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923667968:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769075884.069016:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075884.069052:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923668096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075884.069069:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075884.069074:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075884.069077:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075884.069081:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2573 00000100:00080000:0.0:1769075884.069112:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075884.069118:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075884.069125:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075884.069140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075884.069147:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923668224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075884.069162:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075884.069167:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075884.069171:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075884.069187:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2573 00000100:00080000:0.0:1769075884.069193:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075884.069198:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075884.069219:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075884.069222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075884/real 1769075884] req@ffff969e06a60700 x1855007923667968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075884.069246:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923668352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075884.069256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923667968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075884.069298:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075884.069321:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075884.069330:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075884.069334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075884.069354:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2573 00000100:00080000:1.0:1769075884.069356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075884.069372:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075884.069382:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923667968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075884.069389:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075884.069405:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075884.069424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075884.069445:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923668480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075884.069483:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075884.069506:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075884/real 1769075884] req@ffff969e06a60000 x1855007923668096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075884.069523:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923668096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075884.069534:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075884.069538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075884.069542:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075884.069550:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075884.069573:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075884.069637:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075884/real 1769075884] req@ffff969e06a63800 x1855007923668224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075884.069665:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923668224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075884.069693:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075884.069700:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075884.069708:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075884.069730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075884.069763:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075884.069778:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075884/real 1769075884] req@ffff969e06a60a80 x1855007923668352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075884.069799:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923668352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075884.069810:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075884.069814:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075884.069818:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075884.069839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075884.069858:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075884.069869:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075884/real 1769075884] req@ffff969e06a61f80 x1855007923668480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075884.069883:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923668480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075884.069893:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075884.069897:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075884.069900:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075884.388747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075885.092829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075885.412666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075886.116890:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075886.436716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075887.140778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075887.460677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075888.164742:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075888.484685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075889.188701:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075889.188716:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075889.188722:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075889.188735:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2578 00000100:00080000:0.0:1769075889.188748:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075889.188757:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075889.188797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075889.188816:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923668608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075889.188871:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075889.188887:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075889.188890:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075889.188896:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2578 00000100:00080000:0.0:1769075889.188901:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075889.188908:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075889.188920:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075889.188941:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923668736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075889.188947:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923668608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075889.188992:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075889.188998:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075889.189001:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075889.189007:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2578 00000100:00080000:0.0:1769075889.189013:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075889.189018:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075889.189025:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075889.189041:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923668864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075889.189062:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075889.189067:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075889.189073:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769075889.189075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075889.189081:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2578 00000100:00080000:0.0:1769075889.189088:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075889.189094:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075889.189111:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075889.189131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075889/real 1769075889] req@ffff969e06a62680 x1855007923668608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075889.189132:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923668992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075889.189232:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075889.189236:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075889.189238:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075889.189243:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2578 00000100:00080000:0.0:1769075889.189252:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075889.189259:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075889.189266:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075889.189274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923668608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075889.189283:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923669120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075889.189297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075889.189308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075889.189314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923668608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075889.189346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075889.189376:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075889.189390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075889/real 1769075889] req@ffff969e06a62d80 x1855007923668736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075889.189405:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923668736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075889.189414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075889.189418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075889.189422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075889.189428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075889.189445:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075889.189461:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075889/real 1769075889] req@ffff969e06a61500 x1855007923668864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075889.189485:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923668864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075889.189494:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075889.189498:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075889.189502:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075889.189508:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075889.189545:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075889.189555:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075889/real 1769075889] req@ffff969e06a61180 x1855007923668992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075889.189566:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923668992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075889.189575:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075889.189578:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075889.189581:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923668992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075889.189604:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075889.189651:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075889.189665:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075889/real 1769075889] req@ffff969e06a60380 x1855007923669120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075889.189677:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923669120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075889.189702:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075889.189706:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075889.189710:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075889.508735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075890.212817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075890.532789:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075891.236856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075891.556692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075892.260868:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075892.580737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075893.284823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075893.604756:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075894.308718:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075894.308731:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075894.308737:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075894.308748:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2583 00000100:00080000:0.0:1769075894.308762:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075894.308773:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769075894.308823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769075894.308827:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923669248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075894.308890:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075894.308912:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075894.308915:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075894.308949:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2583 00000100:00080000:0.0:1769075894.308955:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075894.308961:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075894.308973:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075894.309008:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923669376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075894.309022:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075894.309026:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075894.309030:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075894.309033:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923669248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075894.309038:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2583 00000100:00080000:0.0:1769075894.309047:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075894.309051:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075894.309056:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075894.309072:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923669504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075894.309081:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075894.309086:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075894.309089:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075894.309170:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2583 00000800:00000100:1.0:1769075894.309181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075894.309184:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075894.309199:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075894.309237:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075894.309244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075894/real 1769075894] req@ffff969e06a61880 x1855007923669248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075894.309270:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923669632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075894.309276:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923669248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075894.309330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075894.309332:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075894.309344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075894.309345:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075894.309353:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075894.309357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923669248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075894.309365:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2583 00000100:00080000:0.0:1769075894.309376:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075894.309399:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075894.309404:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075894.309416:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075894.309421:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923669760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075894.309470:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075894.309489:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075894/real 1769075894] req@ffff969e06a63100 x1855007923669376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075894.309503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923669376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075894.309541:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075894.309546:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075894.309550:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075894.309558:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075894.309581:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075894.309593:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075894/real 1769075894] req@ffff969e06a60380 x1855007923669504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075894.309606:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923669504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075894.309633:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075894.309636:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075894.309640:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075894.309648:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075894.309704:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075894.309737:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075894/real 1769075894] req@ffff969e06a61180 x1855007923669632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075894.309752:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923669632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075894.309762:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075894.309766:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075894.309779:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075894.309801:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075894.309827:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075894.309855:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075894/real 1769075894] req@ffff969e06a61500 x1855007923669760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075894.309866:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923669760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075894.309874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075894.309877:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075894.309880:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923669760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075894.628737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075895.332781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075895.652747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075896.356766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075896.676711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075897.380809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075897.700722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769075898.404777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075898.724694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075899.428721:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075899.428731:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075899.428735:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075899.428743:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2588 00000100:00080000:0.0:1769075899.428754:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075899.428760:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075899.428794:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923669888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075899.428818:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769075899.428818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075899.428831:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075899.428834:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075899.428837:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2588 00000100:00100000:1.0:1769075899.428856:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923669888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075899.428857:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075899.428865:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075899.428874:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075899.428887:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923670016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075899.428896:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075899.428898:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075899.428900:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075899.428902:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2588 00000100:00080000:0.0:1769075899.428905:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075899.428908:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075899.428911:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075899.428919:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923670144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075899.428925:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075899.428928:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075899.428929:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075899.428932:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2588 00000100:00080000:0.0:1769075899.428935:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075899.428938:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075899.428941:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075899.428950:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075899.428951:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923670272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075899.428975:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075899.428978:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075899.428980:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075899.428985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670144:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769075899.428986:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2588 00000100:00080000:0.0:1769075899.428996:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075899.428999:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075899.429003:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075899.429033:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923670400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075899.429080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075899.429112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075899.748686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075900.452769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075900.772680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075901.476907:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075901.796782:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075902.500851:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769075902.501003:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769075902.501064:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769075902.501088:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075902.501148:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075902.501165:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769075902.501181:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769075902.501183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075899/real 1769075902] req@ffff969e06a62680 x1855007923670016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769075902.501209:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769075902.501252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923670016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.501283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.501289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075902.501294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075902.501303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075899/real 1769075902] req@ffff969e06a61f80 x1855007923670144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.501326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923670144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.501336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.501341:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075902.501345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075902.501353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075899/real 1769075902] req@ffff969e06a62d80 x1855007923669888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.501368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923669888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.501377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.501380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075902.501412:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923669888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075902.501448:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075899/real 1769075902] req@ffff969e06a60a80 x1855007923670272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.501468:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923670272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.501479:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.501483:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075902.501487:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075902.501497:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075899/real 1769075902] req@ffff969e06a63800 x1855007923670400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075902.501510:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923670400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075902.501521:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075902.501525:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075902.501529:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075902.820754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769075903.524763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075903.844717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075904.548772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075904.548791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075904.548800:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075904.548817:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2593 00000100:00080000:2.0:1769075904.548837:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075904.548848:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075904.548901:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923670528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769075904.548909:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075904.548976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075904.548981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075904.548984:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075904.548991:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2593 00000100:00080000:2.0:1769075904.548997:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075904.549018:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075904.549037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923670528:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769075904.549049:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075904.549072:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923670656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075904.549089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075904.549093:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075904.549096:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075904.549102:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2593 00000100:00080000:2.0:1769075904.549109:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075904.549117:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769075904.549119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769075904.549126:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075904.549145:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923670784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075904.549159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075904.549164:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769075904.549170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075904/real 1769075904] req@ffff969e0a46ea00 x1855007923670528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075904.549170:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075904.549181:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2593 00000100:00080000:2.0:1769075904.549191:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075904.549197:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075904.549204:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075904.549224:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923670912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075904.549238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923670528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075904.549239:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075904.549251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075904.549261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075904.549266:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075904.549272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769075904.549279:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2593 00000100:00100000:1.0:1769075904.549286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923670528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769075904.549289:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075904.549302:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075904.549309:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075904.549326:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923671040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075904.549349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075904.549389:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075904.549406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075904/real 1769075904] req@ffff969e0a46df80 x1855007923670656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075904.549421:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923670656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075904.549447:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075904.549452:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075904.549456:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075904.549464:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075904.549491:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075904.549506:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075904/real 1769075904] req@ffff969e0a46f480 x1855007923670784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075904.549522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923670784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075904.549557:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075904.549563:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075904.549568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075904.549576:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075904.549643:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075904.549659:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075904/real 1769075904] req@ffff969e0a46d500 x1855007923670912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075904.549683:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923670912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075904.549695:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075904.549701:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075904.549706:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923670912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075904.549714:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075904.549740:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075904.549756:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075904/real 1769075904] req@ffff969e0a46c380 x1855007923671040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075904.549774:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923671040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075904.549783:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075904.549786:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075904.549800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075904.868696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075905.572745:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075905.892715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075906.596848:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075906.916706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075907.620755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075907.940715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075908.644793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075908.964729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075909.668735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075909.668769:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075909.668776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075909.668780:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075909.668788:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2598 00000100:00080000:3.0:1769075909.668797:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075909.668803:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075909.668891:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923671168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075909.668930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075909.668936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075909.668939:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075909.668946:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2598 00000100:00080000:3.0:1769075909.668953:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075909.668960:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075909.668970:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923671168:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075909.668994:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075909.669021:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923671296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075909.669040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075909.669044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075909.669046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075909.669072:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075909.669078:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2598 00000100:00080000:3.0:1769075909.669085:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075909.669091:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075909.669097:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075909.669111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075909/real 1769075909] req@ffff969e0a7fe300 x1855007923671168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075909.669116:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923671424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075909.669149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075909.669157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923671168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075909.669158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075909.669181:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075909.669195:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2598 00000100:00080000:1.0:1769075909.669196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075909.669206:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075909.669207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075909.669218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923671168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075909.669220:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075909.669236:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075909.669270:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923671552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075909.669275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671296:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075909.669292:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075909.669296:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075909.669306:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075909.669313:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2598 00000100:00080000:3.0:1769075909.669319:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075909.669321:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075909.669337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075909/real 1769075909] req@ffff969e0a7fed80 x1855007923671296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075909.669342:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075909.669349:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075909.669364:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923671296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075909.669366:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923671680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075909.669386:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075909.669393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075909.669398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075909.669404:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075909.669431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075909.669440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075909/real 1769075909] req@ffff969e0a7fe680 x1855007923671424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075909.669449:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923671424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075909.669455:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075909.669458:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075909.669461:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075909.669477:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075909.669499:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075909.669511:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075909/real 1769075909] req@ffff969e0a7fc700 x1855007923671552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075909.669525:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923671552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075909.669533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075909.669535:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075909.669544:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075909.669549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075909.669565:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075909.669573:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075909/real 1769075909] req@ffff969e0a7fd880 x1855007923671680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075909.669581:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923671680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075909.669587:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075909.669614:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075909.669618:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075909.988721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075910.692706:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075911.012673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075911.716675:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075912.036679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075912.740743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075913.060665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075913.764692:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075914.084721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075914.788670:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075914.788680:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075914.788684:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075914.788693:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2603 00000100:00080000:3.0:1769075914.788703:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1769075914.788708:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075914.788709:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075914.788742:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923671808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075914.788779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075914.788781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075914.788783:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075914.788786:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2603 00000100:00080000:3.0:1769075914.788788:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075914.788791:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075914.788799:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075914.788814:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923671936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075914.788815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923671808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075914.788856:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075914.788860:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075914.788862:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075914.788868:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2603 00000100:00080000:3.0:1769075914.788873:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075914.788878:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075914.788882:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075914.788890:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075914.788907:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075914/real 1769075914] req@ffff969e0a7fea00 x1855007923671808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075914.788909:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923672064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075914.788930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075914.788940:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075914.788943:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075914.788949:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2603 00000100:00080000:3.0:1769075914.788953:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075914.788954:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923671808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075914.788963:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769075914.788967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075914.788972:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075914.788974:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075914.788980:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923671808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075914.788990:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923672192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075914.789004:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671936:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075914.789005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075914.789014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075914.789016:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075914.789021:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2603 00000800:00000100:1.0:1769075914.789030:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075914.789037:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075914.789046:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075914.789047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075914/real 1769075914] req@ffff969e0a7fd880 x1855007923671936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075914.789057:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075914.789061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923671936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075914.789069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075914.789072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769075914.789077:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923672320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075914.789078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923671936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075914.789088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075914.789110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075914.789118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075914/real 1769075914] req@ffff969e0a7fc700 x1855007923672064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075914.789126:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923672064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075914.789141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075914.789144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075914.789146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075914.789156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075914.789170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075914.789176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075914/real 1769075914] req@ffff969e0a7fe680 x1855007923672192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075914.789183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923672192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075914.789189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075914.789191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075914.789193:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075914.789196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075914.789205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075914.789210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075914/real 1769075914] req@ffff969e0a7fed80 x1855007923672320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075914.789217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923672320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075914.789222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075914.789224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075914.789232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075915.108665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075915.812728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075916.132686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075916.836774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075917.156804:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075917.860811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075918.180784:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075918.884752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075919.204732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769075919.908737:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075919.908757:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075919.908764:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075919.908776:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2609 00010000:00010000:1.0:1769075919.908779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075919.908790:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075919.908798:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075919.908845:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923672448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075919.908871:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075919.908874:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075919.908875:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075919.908879:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2609 00000100:00080000:2.0:1769075919.908881:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075919.908885:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075919.908894:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075919.908904:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923672448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075919.908906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923672576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075919.908944:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075919.908946:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075919.908948:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075919.908952:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2609 00000100:00080000:2.0:1769075919.908956:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075919.908958:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075919.908962:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075919.908974:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923672704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075919.908981:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075919.908983:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075919.908986:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075919.908989:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075919.908993:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2609 00000100:00080000:2.0:1769075919.908996:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075919.908998:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075919.909002:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075919.909011:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923672832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075919.909017:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075919.909023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075919/real 1769075919] req@ffff969e0a46f800 x1855007923672448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075919.909023:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075919.909025:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075919.909032:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2609 00000100:00080000:2.0:1769075919.909043:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075919.909048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923672448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075919.909052:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075919.909064:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075919.909065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075919.909074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075919.909077:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923672448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075919.909085:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923672960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075919.909113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075919.909131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075919.909140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075919/real 1769075919] req@ffff969e0a46f100 x1855007923672576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075919.909148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923672576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075919.909162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075919.909164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075919.909167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075919.909171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075919.909186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075919.909193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075919/real 1769075919] req@ffff969e0a46c380 x1855007923672704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075919.909200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923672704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075919.909206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075919.909207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075919.909209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075919.909213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075919.909224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075919.909230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075919/real 1769075919] req@ffff969e0a46d500 x1855007923672832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075919.909238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923672832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075919.909243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075919.909245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075919.909253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075919.909257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075919.909271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075919.909278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075919/real 1769075919] req@ffff969e0a46f480 x1855007923672960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075919.909286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923672960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075919.909292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075919.909294:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075919.909296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923672960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075920.228685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075920.932800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075921.252737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075921.956785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075922.276703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075922.980782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075923.300749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075924.004722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075924.324700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075925.028733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075925.028738:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075925.028749:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075925.028754:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075925.028765:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2614 00000100:00080000:0.0:1769075925.028775:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075925.028781:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075925.028815:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923673088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075925.028842:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075925.028845:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075925.028861:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075925.028866:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2614 00000100:00080000:0.0:1769075925.028871:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075925.028876:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075925.028885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923673088:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769075925.028886:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075925.028908:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923673216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075925.028920:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075925.028923:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075925.028925:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075925.028929:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2614 00000100:00080000:0.0:1769075925.028931:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075925.028936:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075925.028940:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075925.028981:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923673344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075925.028991:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075925.028992:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075925.029000:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075925.029003:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075925.029009:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2614 00000100:00080000:0.0:1769075925.029015:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075925.029021:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075925.029027:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075925.029042:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075925/real 1769075925] req@ffff969e06a60000 x1855007923673088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075925.029051:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923673472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075925.029088:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075925.029090:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075925.029092:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075925.029097:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2614 00000100:00080000:0.0:1769075925.029109:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075925.029139:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075925.029146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923673088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075925.029149:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075925.029174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769075925.029177:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923673600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075925.029192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075925.029212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923673088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075925.029253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075925.029274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075925.029284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075925/real 1769075925] req@ffff969e06a60700 x1855007923673216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075925.029293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923673216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075925.029299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075925.029302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075925.029304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075925.029309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075925.029331:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075925.029337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075925/real 1769075925] req@ffff969e06a63800 x1855007923673344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075925.029344:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923673344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075925.029359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075925.029362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075925.029365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075925.029370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075925.029393:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075925.029404:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075925/real 1769075925] req@ffff969e06a60a80 x1855007923673472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075925.029417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923673472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075925.029433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075925.029438:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075925.029441:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075925.029448:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075925.029478:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075925.029486:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075925/real 1769075925] req@ffff969e06a62d80 x1855007923673600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075925.029494:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923673600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075925.029500:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075925.029502:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075925.029510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075925.348719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075926.052797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075926.372694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075927.076768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075927.396766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075928.100753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075928.420684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075929.124735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075929.444714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075930.148720:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075930.148728:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075930.148733:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075930.148742:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2619 00000100:00080000:3.0:1769075930.148752:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075930.148758:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:1.0:1769075930.148777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769075930.148794:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923673728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075930.148813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075930.148816:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075930.148819:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075930.148822:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2619 00000100:00080000:3.0:1769075930.148825:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075930.148828:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075930.148836:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075930.148845:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923673856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075930.148853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075930.148859:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923673728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075930.148875:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075930.148876:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075930.148880:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2619 00000100:00080000:3.0:1769075930.148883:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075930.148885:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075930.148889:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075930.148899:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923673984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075930.148906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075930.148909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075930.148910:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075930.148913:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2619 00000100:00080000:3.0:1769075930.148915:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075930.148917:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075930.148920:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075930.148927:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923674112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075930.148933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075930.148935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075930.148937:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075930.148939:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2619 00000100:00080000:3.0:1769075930.148943:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075930.148946:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075930.148954:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075930.148963:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923674240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075930.148982:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075930.149035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075930/real 1769075930] req@ffff969e0a7fe300 x1855007923673728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075930.149060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923673728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075930.149074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075930.149080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075930.149086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923673728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075930.149120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075930.149151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075930.149168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075930/real 1769075930] req@ffff969e0a7fc380 x1855007923673856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075930.149197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923673856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075930.149220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075930.149224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075930.149228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075930.149236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075930.149265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075930.149278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075930/real 1769075930] req@ffff969e0a7fdf80 x1855007923673984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075930.149302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923673984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075930.149312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075930.149317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075930.149321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923673984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075930.149329:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075930.149355:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075930.149369:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075930/real 1769075930] req@ffff969e0a7fd180 x1855007923674112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075930.149384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923674112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075930.149393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075930.149397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075930.149408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075930.149416:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075930.149443:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075930.149457:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075930/real 1769075930] req@ffff969e0a7fce00 x1855007923674240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075930.149470:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923674240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075930.149480:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075930.149484:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075930.149488:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075930.468679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075931.172712:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075931.492702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075932.196767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075932.516676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075933.220703:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075933.540672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075934.244793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075934.564682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075935.268723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075935.268771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075935.268788:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075935.268795:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075935.268811:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2624 00000100:00080000:2.0:1769075935.268826:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075935.268838:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075935.268895:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923674368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075935.268946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075935.268953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075935.268957:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075935.268967:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2624 00000100:00080000:2.0:1769075935.268974:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075935.268984:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075935.268994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923674368:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769075935.269006:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075935.269042:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923674496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075935.269063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075935.269092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075935.269107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075935.269112:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075935.269124:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2624 00000100:00080000:2.0:1769075935.269135:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075935.269144:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075935.269154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075935/real 1769075935] req@ffff969e0a46df80 x1855007923674368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769075935.269158:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075935.269194:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923674624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075935.269230:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075935.269233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923674368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769075935.269240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075935.269249:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075935.269261:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2624 00000100:00080000:1.0:1769075935.269262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769075935.269274:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075935.269275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769075935.269289:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075935.269291:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923674368:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769075935.269307:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075935.269328:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923674752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075935.269341:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674496:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769075935.269345:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075935.269351:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075935.269354:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075935.269363:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2624 00000100:00080000:2.0:1769075935.269368:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769075935.269374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769075935.269375:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075935.269393:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075935/real 1769075935] req@ffff969e0a46ea00 x1855007923674496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769075935.269397:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075935.269440:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923674496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075935.269450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075935.269454:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769075935.269461:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923674880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075935.269469:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075935.269502:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075935.269539:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075935.269550:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075935/real 1769075935] req@ffff969e0a46ca80 x1855007923674624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075935.269570:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923674624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075935.269623:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075935.269628:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075935.269632:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075935.269648:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075935.269667:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075935.269675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075935/real 1769075935] req@ffff969e0a46d880 x1855007923674752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075935.269684:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923674752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075935.269690:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075935.269693:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075935.269695:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075935.269699:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075935.269709:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075935.269715:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075935/real 1769075935] req@ffff969e0a46ed80 x1855007923674880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075935.269722:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923674880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075935.269728:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075935.269730:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075935.269737:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923674880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075935.588723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075936.292776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075936.612713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075937.316701:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075937.636688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075938.340740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075938.660739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075939.364772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075939.684667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075940.388714:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1769075940.388717:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075940.388727:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075940.388733:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075940.388746:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2629 00000100:00080000:0.0:1769075940.388762:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075940.388772:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075940.388833:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923675008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075940.388860:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075940.388863:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075940.388865:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075940.388869:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2629 00000100:00080000:0.0:1769075940.388874:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075940.388891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923675008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075940.388902:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075940.388913:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075940.388927:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923675136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075940.388936:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075940.388939:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075940.388941:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075940.388943:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2629 00000100:00080000:0.0:1769075940.388958:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075940.388961:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075940.388965:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075940.388974:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923675264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075940.388979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075940.388987:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075940.388991:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075940.388992:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075940.388996:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2629 00000100:00080000:0.0:1769075940.388998:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075940.389001:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075940.389005:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075940.389013:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923675392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075940.389020:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075940.389029:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075940.389032:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075940.389032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075940/real 1769075940] req@ffff969e06a61f80 x1855007923675008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075940.389039:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2629 00000100:00080000:0.0:1769075940.389045:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075940.389052:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075940.389058:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075940.389067:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923675520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075940.389071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923675008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075940.389092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075940.389097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075940.389102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923675008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075940.389129:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075940.389150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075940.389164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075940/real 1769075940] req@ffff969e06a62680 x1855007923675136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075940.389178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923675136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075940.389202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075940.389215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075940.389218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075940.389223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075940.389242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075940.389255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075940/real 1769075940] req@ffff969e06a61500 x1855007923675264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075940.389264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923675264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075940.389270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075940.389272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075940.389274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075940.389278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075940.389291:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075940.389297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075940/real 1769075940] req@ffff969e06a61180 x1855007923675392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075940.389308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923675392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075940.389318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075940.389356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075940.389367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075940.389374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075940.389392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075940.389400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075940/real 1769075940] req@ffff969e06a60380 x1855007923675520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075940.389409:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923675520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075940.389416:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075940.389419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075940.389421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075940.708736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075941.412811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075941.732671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075942.436736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075942.756744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075943.460776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075943.780695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075944.484791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075944.804728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075945.508753:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075945.508763:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075945.508768:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075945.508778:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2634 00000100:00080000:0.0:1769075945.508789:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1769075945.508796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075945.508798:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075945.508841:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923675648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075945.508874:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075945.508877:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075945.508879:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075945.508883:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2634 00000100:00080000:0.0:1769075945.508887:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075945.508892:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075945.508903:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075945.508914:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923675776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075945.508923:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923675648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075945.508961:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075945.508966:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075945.508970:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075945.508977:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2634 00000100:00080000:0.0:1769075945.508981:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075945.508986:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075945.508993:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075945.509014:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923675904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075945.509028:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075945.509035:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075945.509038:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075945.509044:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2634 00000800:00000100:1.0:1769075945.509052:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769075945.509059:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075945.509072:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075945.509083:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075945.509101:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923676032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075945.509103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075945/real 1769075945] req@ffff969e06a63100 x1855007923675648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075945.509159:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075945.509165:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075945.509173:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075945.509196:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2634 00000100:00100000:1.0:1769075945.509207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923675648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075945.509226:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769075945.509231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075945.509261:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075945.509278:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075945.509279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075945.509297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923675648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769075945.509317:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923676160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075945.509334:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075945.509364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075945.509379:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075945/real 1769075945] req@ffff969e06a61880 x1855007923675776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075945.509393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923675776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075945.509401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075945.509404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075945.509408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075945.509414:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075945.509427:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075945.509450:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075945/real 1769075945] req@ffff969e06a60380 x1855007923675904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075945.509479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923675904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075945.509499:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075945.509504:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075945.509508:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923675904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075945.509519:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075945.509548:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075945.509556:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075945/real 1769075945] req@ffff969e06a61180 x1855007923676032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075945.509565:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923676032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075945.509571:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075945.509573:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075945.509575:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075945.509616:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075945.509630:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075945.509643:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075945/real 1769075945] req@ffff969e06a61500 x1855007923676160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075945.509653:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923676160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075945.509660:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075945.509662:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075945.509670:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075945.828723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075946.532746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075946.852725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075947.556747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075947.876705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075948.580751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075948.900701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075949.604736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075949.924696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769075950.628744:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075950.628756:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1769075950.628757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075950.628771:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075950.628784:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2639 00000100:00080000:0.0:1769075950.628795:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075950.628802:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075950.628843:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923676288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075950.628868:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075950.628872:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075950.628874:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075950.628878:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2639 00000100:00080000:0.0:1769075950.628881:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075950.628885:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075950.628895:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075950.628909:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923676416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075950.628919:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923676288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075950.628924:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075950.628947:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075950.628949:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075950.628955:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2639 00000100:00080000:0.0:1769075950.628960:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075950.628964:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075950.628980:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075950.628991:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075950.628998:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923676544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075950.629011:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075950.629014:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075950.629016:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075950.629021:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2639 00000100:00080000:0.0:1769075950.629026:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075950.629031:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075950.629035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075950/real 1769075950] req@ffff969e06a62680 x1855007923676288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769075950.629039:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075950.629062:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923676672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075950.629066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923676288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075950.629092:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075950.629097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075950.629098:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769075950.629104:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769075950.629109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769075950.629113:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2639 00000100:00080000:0.0:1769075950.629118:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075950.629119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923676288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075950.629125:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075950.629139:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075950.629169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769075950.629174:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923676800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075950.629209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075950.629224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075950/real 1769075950] req@ffff969e06a61f80 x1855007923676416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075950.629237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923676416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075950.629254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075950.629257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075950.629261:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075950.629267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075950.629287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075950.629296:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075950/real 1769075950] req@ffff969e06a62d80 x1855007923676544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075950.629306:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923676544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075950.629313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075950.629315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075950.629318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075950.629323:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075950.629337:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075950.629344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075950/real 1769075950] req@ffff969e06a60a80 x1855007923676672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075950.629354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923676672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075950.629361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075950.629364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075950.629374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075950.629392:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075950.629421:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075950.629437:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075950/real 1769075950] req@ffff969e06a63800 x1855007923676800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075950.629454:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923676800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075950.629468:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075950.629473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075950.629478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923676800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075950.948788:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075951.652773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075951.972792:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075952.676831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075952.996748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075953.700744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075954.020776:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075954.724744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075955.044709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075955.748778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075955.748873:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075955.748884:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075955.748890:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075955.748909:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2644 00000100:00080000:3.0:1769075955.748923:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075955.748932:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075955.748989:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923676928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075955.749036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075955.749041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075955.749044:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075955.749049:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2644 00000100:00080000:3.0:1769075955.749053:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075955.749058:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075955.749095:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075955.749133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923676928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075955.749134:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923677056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075955.749163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075955.749167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075955.749170:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075955.749176:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2644 00000100:00080000:3.0:1769075955.749203:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075955.749210:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075955.749219:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075955.749234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075955.749242:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923677184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075955.749260:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075955.749265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075955.749268:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075955.749274:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2644 00000100:00080000:3.0:1769075955.749281:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075955.749287:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075955.749296:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075955/real 1769075955] req@ffff969e0a7fca80 x1855007923676928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769075955.749297:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075955.749324:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923677312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075955.749341:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075955.749356:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075955.749363:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075955.749375:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2644 00000100:00080000:3.0:1769075955.749385:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075955.749390:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075955.749397:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075955.749406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923676928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075955.749416:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923677440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075955.749443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075955.749455:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075955.749470:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923676928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075955.749515:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075955.749548:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075955.749564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075955/real 1769075955] req@ffff969e0a7fc000 x1855007923677056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075955.749579:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923677056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075955.749621:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075955.749626:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075955.749631:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075955.749639:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075955.749662:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075955.749675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075955/real 1769075955] req@ffff969e0a7fd500 x1855007923677184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075955.749690:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923677184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075955.749718:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075955.749722:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075955.749727:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075955.749735:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075955.749761:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075955.749773:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075955/real 1769075955] req@ffff969e0a7fdc00 x1855007923677312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075955.749787:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923677312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075955.749798:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075955.749801:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075955.749804:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075955.749811:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075955.749829:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075955.749839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075955/real 1769075955] req@ffff969e0a7ff800 x1855007923677440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075955.749853:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923677440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075955.749863:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075955.749866:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075955.749879:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075956.068740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075956.772797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075957.092757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075957.796771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075958.116772:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075958.820735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075959.140767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075959.844774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075960.164709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075960.868721:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075960.868737:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075960.868743:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075960.868757:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2650 00000100:00080000:3.0:1769075960.868772:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1769075960.868783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075960.868783:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075960.868841:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923677568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075960.868877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075960.868881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075960.868883:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075960.868888:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2650 00000100:00080000:3.0:1769075960.868893:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075960.868897:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075960.868910:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075960.868926:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923677696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075960.868940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769075960.868944:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923677568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075960.868945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075960.868950:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075960.868961:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2650 00000100:00080000:3.0:1769075960.868965:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075960.868969:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075960.869024:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075960.869041:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923677824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075960.869058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075960.869064:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075960.869067:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075960.869074:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2650 00000100:00080000:3.0:1769075960.869084:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075960.869090:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075960.869099:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075960.869115:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075960.869125:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923677952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075960.869145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075960.869149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075960.869152:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075960.869157:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2650 00000100:00080000:3.0:1769075960.869163:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075960.869168:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075960.869175:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075960.869176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075960/real 1769075960] req@ffff969e0a7ffb80 x1855007923677568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075960.869196:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923678080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075960.869209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923677568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075960.869232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075960.869239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075960.869244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923677568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075960.869278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075960.869317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075960.869340:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075960/real 1769075960] req@ffff969e0a7ff800 x1855007923677696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075960.869363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923677696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075960.869387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075960.869392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075960.869396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075960.869403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075960.869430:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075960.869443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075960/real 1769075960] req@ffff969e0a7fdc00 x1855007923677824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075960.869467:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923677824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075960.869477:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075960.869480:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075960.869483:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075960.869490:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075960.869510:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075960.869525:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075960/real 1769075960] req@ffff969e0a7fd500 x1855007923677952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075960.869544:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923677952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075960.869558:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075960.869564:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075960.869581:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923677952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075960.869626:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075960.869659:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075960.869678:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075960/real 1769075960] req@ffff969e0a7fc000 x1855007923678080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075960.869698:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923678080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075960.869713:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075960.869719:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075960.869725:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075961.188652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075961.892734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075962.212751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075962.916779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075963.236782:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075963.940800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075964.260703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075964.964757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075965.284727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075965.988745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075965.988759:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1769075965.988784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075965.988789:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075965.988803:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2655 00000100:00080000:3.0:1769075965.988829:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075965.988839:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075965.988883:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923678208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075965.988954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075965.988959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075965.988962:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075965.988968:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2655 00000100:00080000:3.0:1769075965.988973:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075965.988979:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075965.988992:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075965.989014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923678208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075965.989016:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923678336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075965.989042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075965.989046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075965.989049:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075965.989055:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2655 00000100:00080000:3.0:1769075965.989059:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075965.989064:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075965.989071:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075965.989086:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923678464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075965.989117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075965.989122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075965.989125:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075965.989132:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2655 00000100:00080000:3.0:1769075965.989141:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075965.989146:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075965.989154:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075965.989166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075965.989177:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923678592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075965.989196:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075965.989202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075965.989206:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075965.989217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678464:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769075965.989218:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2655 00000100:00080000:3.0:1769075965.989232:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075965.989240:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075965.989246:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075965.989267:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923678720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075965.989281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075965.989358:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075966.308731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075967.012746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075967.332683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075968.036773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075968.356710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075969.060784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769075969.060944:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769075969.060981:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769075969.060997:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075969.061034:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075969.061042:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075969.061050:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769075969.061057:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769075969.061087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075965/real 1769075969] req@ffff969e0a7fc380 x1855007923678720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.061113:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923678720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.061125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.061129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075969.061133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075969.061140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075965/real 1769075969] req@ffff969e0a7fdf80 x1855007923678592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.061149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923678592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.061155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.061157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075969.061160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075969.061166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075965/real 1769075969] req@ffff969e0a7fce00 x1855007923678336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.061198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923678336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.061205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.061207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075969.061210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075969.061215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075965/real 1769075969] req@ffff969e0a7fd180 x1855007923678464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.061224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923678464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.061230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.061232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075969.061234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075969.061239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075965/real 1769075969] req@ffff969e0a7fca80 x1855007923678208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075969.061251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923678208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075969.061261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075969.061264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075969.061268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923678208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075969.380689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075970.084735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075970.404802:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075971.108751:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075971.108776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075971.108785:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769075971.108799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075971.108838:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2660 00000100:00080000:3.0:1769075971.108900:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075971.108912:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075971.108963:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923678848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075971.109001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075971.109005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075971.109008:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075971.109013:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2660 00000100:00080000:3.0:1769075971.109017:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075971.109022:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075971.109035:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075971.109053:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923678976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075971.109062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923678848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075971.109105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075971.109112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075971.109116:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075971.109136:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2660 00000100:00080000:3.0:1769075971.109144:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075971.109153:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075971.109163:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075971.109183:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923679104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075971.109201:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075971.109203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075971.109228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075971.109233:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075971.109243:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2660 00000100:00080000:3.0:1769075971.109253:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769075971.109255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075971/real 1769075971] req@ffff969e0a7fe300 x1855007923678848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075971.109266:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075971.109299:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075971.109312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923678848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075971.109326:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923679232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769075971.109337:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769075971.109350:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075971.109353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075971.109363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923678848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769075971.109374:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075971.109380:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075971.109388:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2660 00000100:00080000:3.0:1769075971.109413:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075971.109420:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075971.109429:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075971.109432:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769075971.109449:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923679360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075971.109473:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075971.109498:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075971/real 1769075971] req@ffff969e0a7fed80 x1855007923678976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075971.109518:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923678976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075971.109533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075971.109538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075971.109543:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923678976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075971.109552:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075971.109604:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075971.109655:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075971/real 1769075971] req@ffff969e0a7fe680 x1855007923679104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075971.109678:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923679104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075971.109690:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075971.109695:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075971.109700:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075971.109709:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075971.109740:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075971.109759:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075971/real 1769075971] req@ffff969e0a7fc700 x1855007923679232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075971.109777:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923679232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075971.109790:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075971.109795:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075971.109801:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075971.109857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075971.109893:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075971.109911:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075971/real 1769075971] req@ffff969e0a7fd880 x1855007923679360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075971.109957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923679360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769075996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075971.109974:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075971.109978:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075971.109983:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075971.428725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075972.132889:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075972.452697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075973.156766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075973.476721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075974.180779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075974.500685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075975.204728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075975.524708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075976.228760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769075976.228789:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075976.228802:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769075976.228809:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075976.228827:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2665 00000100:00080000:0.0:1769075976.228854:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075976.228864:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769075976.228912:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923679488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075976.228951:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075976.228958:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769075976.228961:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075976.228970:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2665 00000100:00080000:0.0:1769075976.228977:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075976.228984:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075976.229017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923679488:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769075976.229018:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769075976.229044:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923679616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075976.229064:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075976.229070:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769075976.229073:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075976.229079:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2665 00000100:00080000:0.0:1769075976.229101:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075976.229109:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075976.229118:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769075976.229125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769075976.229139:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923679744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075976.229157:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769075976.229163:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769075976.229166:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075976.229173:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2665 00000100:00100000:1.0:1769075976.229176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075976/real 1769075976] req@ffff969e06a60700 x1855007923679488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769075976.229179:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075976.229186:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769075976.229196:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075976.229209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923679488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769075976.229212:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923679872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769075976.229231:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769075976.229231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769075976.229237:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769075976.229240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075976.229247:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923679488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769075976.229252:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769075976.229275:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2665 00000100:00080000:0.0:1769075976.229284:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769075976.229291:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075976.229295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679616:192.168.204.143@tcp:38:kworker.0 00000008:00100000:0.0:1769075976.229301:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769075976.229322:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923680000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075976.229338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075976.229387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075976/real 1769075976] req@ffff969e06a60000 x1855007923679616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075976.229415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923679616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075976.229426:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075976.229431:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075976.229435:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075976.229444:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075976.229474:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075976.229489:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075976/real 1769075976] req@ffff969e06a63800 x1855007923679744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075976.229503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923679744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075976.229513:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075976.229517:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075976.229521:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075976.229528:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075976.229547:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075976.229558:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075976/real 1769075976] req@ffff969e06a60a80 x1855007923679872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075976.229587:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923679872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075976.229633:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075976.229637:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075976.229642:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923679872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075976.229666:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075976.229694:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075976.229708:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075976/real 1769075976] req@ffff969e06a62d80 x1855007923680000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075976.229754:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923680000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075976.229768:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075976.229773:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075976.229777:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075976.548693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075977.252717:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075977.572675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075978.276716:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075978.596737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075979.300737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075979.620763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075980.324811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075980.644726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075981.348793:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1769075981.348796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075981.348813:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075981.348820:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075981.348835:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2670 00000100:00080000:3.0:1769075981.348852:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075981.348863:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075981.348933:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923680128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075981.349012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075981.349017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075981.349020:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075981.349026:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2670 00000100:00080000:3.0:1769075981.349032:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075981.349038:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075981.349053:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075981.349095:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923680256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075981.349115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075981.349123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075981.349127:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075981.349136:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2670 00000100:00080000:3.0:1769075981.349161:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075981.349169:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075981.349179:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075981.349191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923680128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075981.349207:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923680384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075981.349232:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075981.349240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075981.349244:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075981.349251:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2670 00000100:00080000:3.0:1769075981.349274:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075981.349284:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075981.349296:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075981.349305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075981.349328:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923680512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075981.349354:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075981.349358:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075981.349361:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075981.349368:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2670 00000100:00080000:3.0:1769075981.349376:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075981.349381:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075981.349388:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075981.349392:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075981/real 1769075981] req@ffff969e0a7fea00 x1855007923680128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075981.349406:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923680640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075981.349487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923680128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075981.349513:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075981.349523:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075981.349531:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923680128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075981.349547:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075981.349612:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075981.349730:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075981/real 1769075981] req@ffff969e0a7fd880 x1855007923680256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075981.349747:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923680256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075981.349760:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075981.349765:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075981.349769:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075981.349807:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075981.349836:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075981.349849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075981/real 1769075981] req@ffff969e0a7fc700 x1855007923680384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075981.349893:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923680384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075981.349904:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075981.349909:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075981.349912:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075981.349920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075981.349944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075981.349956:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075981/real 1769075981] req@ffff969e0a7fe680 x1855007923680512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075981.349970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923680512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075981.349979:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075981.349982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075981.349986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075981.349991:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075981.350008:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075981.350018:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075981/real 1769075981] req@ffff969e0a7fed80 x1855007923680640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075981.350030:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923680640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075981.350056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075981.350060:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075981.350064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075981.668740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075982.372838:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075982.692672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075983.396799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075983.716744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075984.420789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075984.740745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075985.444764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075985.764747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075986.468766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075986.468783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075986.468791:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075986.468808:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2675 00010000:00010000:1.0:1769075986.468815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075986.468839:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075986.468854:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075986.468926:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923680768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075986.468968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075986.469000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075986.469004:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075986.469011:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2675 00000100:00080000:3.0:1769075986.469018:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075986.469028:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075986.469031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923680768:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075986.469044:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075986.469069:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923680896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075986.469085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075986.469090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075986.469093:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075986.469098:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2675 00000100:00080000:3.0:1769075986.469105:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075986.469109:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075986.469114:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075986.469129:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923681024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075986.469142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769075986.469151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769075986.469153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075986.469158:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075986.469165:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2675 00000100:00080000:3.0:1769075986.469172:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075986.469177:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075986.469195:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769075986.469212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075986/real 1769075986] req@ffff969e0a7fe300 x1855007923680768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769075986.469218:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923681152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075986.469265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075986.469271:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075986.469274:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769075986.469277:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923680768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769075986.469283:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2675 00000100:00080000:3.0:1769075986.469293:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075986.469316:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769075986.469319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769075986.469328:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769075986.469330:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075986.469342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923680768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769075986.469363:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923681280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075986.469385:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075986.469431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075986.469464:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075986/real 1769075986] req@ffff969e0a7fca80 x1855007923680896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075986.469492:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923680896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075986.469504:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075986.469508:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075986.469513:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923680896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075986.469521:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075986.469545:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075986.469643:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075986/real 1769075986] req@ffff969e0a7fd180 x1855007923681024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075986.469660:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923681024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075986.469670:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075986.469675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075986.469679:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075986.469687:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075986.469712:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075986.469724:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075986/real 1769075986] req@ffff969e0a7fce00 x1855007923681152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075986.469738:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923681152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075986.469763:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075986.469768:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075986.469772:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075986.469794:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075986.469819:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075986.469832:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075986/real 1769075986] req@ffff969e0a7fdf80 x1855007923681280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075986.469853:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923681280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075986.469864:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075986.469867:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075986.469871:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075986.788693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075987.492770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075987.812709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075988.516812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075988.836717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075989.540838:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075989.860750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075990.564780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075990.884729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769075991.588676:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075991.588686:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769075991.588690:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769075991.588699:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769075991.588717:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2680 00000100:00080000:3.0:1769075991.588730:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075991.588737:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769075991.588769:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923681408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075991.588793:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075991.588796:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769075991.588797:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075991.588800:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2680 00000100:00080000:3.0:1769075991.588819:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075991.588823:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769075991.588833:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923681408:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769075991.588833:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075991.588851:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923681536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075991.588864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075991.588869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769075991.588870:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075991.588873:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2680 00000100:00080000:3.0:1769075991.588876:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075991.588878:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075991.588881:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769075991.588891:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923681664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075991.588897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075991.588900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769075991.588901:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075991.588904:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2680 00000100:00080000:3.0:1769075991.588907:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075991.588910:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075991.588913:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769075991.588921:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923681792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769075991.588937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769075991.588939:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769075991.588940:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769075991.588943:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2680 00000100:00080000:3.0:1769075991.588950:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769075991.588953:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769075991.588957:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769075991.588963:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769075991.588976:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923681920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769075991.589012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075991/real 1769075991] req@ffff969e0a7fc380 x1855007923681408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075991.589029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923681408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075991.589038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075991.589048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075991.589052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923681408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075991.589089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075991.589106:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075991.589115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075991/real 1769075991] req@ffff969e0a7fc000 x1855007923681536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075991.589124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923681536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075991.589130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075991.589133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075991.589135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075991.589156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075991.589168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075991.589175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075991/real 1769075991] req@ffff969e0a7fd500 x1855007923681664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075991.589195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923681664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075991.589201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075991.589204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075991.589206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075991.589210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075991.589225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075991.589232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075991/real 1769075991] req@ffff969e0a7fdc00 x1855007923681792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075991.589240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923681792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075991.589246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075991.589248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075991.589250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075991.589254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075991.589264:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075991.589269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075991/real 1769075991] req@ffff969e0a7ff800 x1855007923681920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075991.589277:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923681920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075991.589291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075991.589294:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075991.589297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923681920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075991.908705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075992.612774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075992.932714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075993.636801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075993.956721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075994.660780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075994.980729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075995.684768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075996.004703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075996.708762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769075996.708773:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075996.708786:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769075996.708792:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075996.708804:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2685 00000100:00080000:2.0:1769075996.708816:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075996.708824:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769075996.708867:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923682048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075996.708904:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075996.708908:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769075996.708911:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075996.708915:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2685 00000100:00080000:2.0:1769075996.708918:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075996.708922:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075996.708931:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769075996.708943:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923682176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075996.708956:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075996.708961:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769075996.708963:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075996.708967:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2685 00000100:00080000:2.0:1769075996.708991:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075996.708998:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075996.709004:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769075996.709016:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923682048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769075996.709032:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923682304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075996.709048:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075996.709052:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769075996.709064:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075996.709070:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2685 00000100:00080000:2.0:1769075996.709074:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075996.709079:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075996.709085:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075996.709098:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923682432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769075996.709108:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769075996.709111:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769075996.709113:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769075996.709119:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2685 00000100:00080000:2.0:1769075996.709128:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769075996.709132:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769075996.709136:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769075996.709146:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923682560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769075996.709160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075996.709236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075996/real 1769075996] req@ffff969e0a46ce00 x1855007923682048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075996.709266:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923682048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075996.709282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075996.709288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075996.709294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923682048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769075996.709306:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075996.709347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075996.709416:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075996/real 1769075996] req@ffff969e0a46e300 x1855007923682176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075996.709434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923682176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075996.709446:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075996.709451:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075996.709456:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075996.709497:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769075996.709524:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075996.709538:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075996/real 1769075996] req@ffff969e0a46ed80 x1855007923682304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075996.709553:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923682304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075996.709564:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075996.709568:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075996.709573:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769075996.709581:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075996.709639:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075996.709652:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075996/real 1769075996] req@ffff969e0a46d880 x1855007923682432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075996.709668:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923682432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075996.709690:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075996.709695:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075996.709699:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075996.709708:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769075996.709735:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769075996.709748:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769075996/real 1769075996] req@ffff969e0a46ca80 x1855007923682560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769075996.709762:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923682560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769075996.709773:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769075996.709777:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769075996.709780:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769075997.028676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075997.732758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075998.052694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075998.756789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769075999.076748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769075999.780739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076000.100683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076000.804733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076001.124696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076001.828726:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076001.828756:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076001.828768:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076001.828775:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076001.828789:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2691 00000100:00080000:3.0:1769076001.828804:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076001.828812:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076001.828860:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923682688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076001.828890:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076001.828893:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076001.828895:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076001.828898:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2691 00000100:00080000:3.0:1769076001.828902:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076001.828906:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076001.828933:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923682688:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076001.828936:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076001.828953:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923682816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076001.828963:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076001.828966:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076001.828967:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076001.828970:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2691 00000100:00080000:3.0:1769076001.828974:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076001.828977:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076001.828980:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076001.828989:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923682944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076001.828996:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076001.828999:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076001.829000:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076001.829003:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2691 00000100:00080000:3.0:1769076001.829005:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076001.829008:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076001.829011:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076001.829018:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923683072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076001.829024:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076001.829026:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076001.829032:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076001.829036:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2691 00000100:00080000:3.0:1769076001.829039:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076001.829042:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076001.829048:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769076001.829050:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076001.829062:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923683200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076001.829097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076001/real 1769076001] req@ffff969e0a7ffb80 x1855007923682688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076001.829115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923682688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076001.829124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076001.829128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076001.829131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923682688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076001.829155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076001.829172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076001.829181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076001/real 1769076001] req@ffff969e0a7ff800 x1855007923682816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076001.829190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923682816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076001.829196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076001.829198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076001.829200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076001.829203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076001.829214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076001.829220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076001/real 1769076001] req@ffff969e0a7fdc00 x1855007923682944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076001.829240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923682944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076001.829246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076001.829249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076001.829251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923682944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076001.829255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076001.829269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076001.829275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076001/real 1769076001] req@ffff969e0a7fd500 x1855007923683072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076001.829283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923683072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076001.829316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076001.829318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076001.829320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076001.829325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076001.829340:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076001.829347:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076001/real 1769076001] req@ffff969e0a7fc000 x1855007923683200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076001.829355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923683200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076001.829371:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076001.829373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076001.829376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076002.148680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076002.852808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076003.172696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076003.876759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076004.196762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076004.900801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076005.220729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076005.924810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076006.244695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076006.948755:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076006.948769:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076006.948775:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076006.948790:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2696 00010000:00010000:1.0:1769076006.948807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076006.948807:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076006.948833:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076006.948893:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923683328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076006.948936:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076006.948940:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076006.948943:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076006.948948:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2696 00000100:00080000:3.0:1769076006.948954:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076006.948960:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076006.948974:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076006.949007:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923683328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076006.949008:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923683456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076006.949101:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076006.949106:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076006.949109:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076006.949116:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2696 00000100:00080000:3.0:1769076006.949124:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076006.949130:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076006.949138:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076006.949159:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923683584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076006.949169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076006.949190:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076006.949195:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076006.949199:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076006.949241:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2696 00000100:00080000:3.0:1769076006.949245:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076006.949251:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076006.949258:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076006.949271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076006/real 1769076006] req@ffff969e0a7fc380 x1855007923683328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076006.949278:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923683712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076006.949341:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076006.949345:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076006.949347:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076006.949356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923683328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076006.949356:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2696 00000100:00080000:3.0:1769076006.949375:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076006.949387:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076006.949400:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076006.949410:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076006.949417:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923683840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076006.949428:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076006.949448:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923683328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076006.949520:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076006.949558:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076006.949653:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076006/real 1769076006] req@ffff969e0a7fdf80 x1855007923683456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076006.949684:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923683456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076006.949697:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076006.949702:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076006.949707:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076006.949717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076006.949749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076006.949765:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076006/real 1769076006] req@ffff969e0a7fce00 x1855007923683584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076006.949780:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923683584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076006.949790:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076006.949795:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076006.949799:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076006.949806:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076006.949827:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076006.949837:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076006/real 1769076006] req@ffff969e0a7fd180 x1855007923683712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076006.949851:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923683712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076006.949877:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076006.949881:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076006.949886:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076006.949894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076006.949930:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076006.949943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076006/real 1769076006] req@ffff969e0a7fca80 x1855007923683840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076006.949957:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923683840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076006.949968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076006.949971:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076006.949975:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923683840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076007.268773:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076007.972830:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076008.292727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076008.996773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076009.316764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076010.020751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076010.340673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076011.044738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076011.364743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076012.068678:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076012.068689:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076012.068695:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769076012.068706:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076012.068707:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2701 00000100:00080000:3.0:1769076012.068721:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076012.068730:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076012.068783:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923683968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076012.068811:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076012.068815:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076012.068817:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076012.068822:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2701 00000100:00080000:3.0:1769076012.068825:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076012.068829:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076012.068840:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076012.068859:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923684096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076012.068860:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923683968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076012.068878:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076012.068898:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076012.068900:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076012.068905:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2701 00000100:00080000:3.0:1769076012.068910:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076012.068915:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076012.068921:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076012.068937:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923684224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076012.068948:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076012.068952:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076012.068954:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076012.068957:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2701 00000100:00080000:3.0:1769076012.068960:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076012.068968:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076012.068969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769076012.068976:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076012.069009:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923684352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076012.069023:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076012.069028:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076012.069031:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076012.069039:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2701 00000100:00100000:1.0:1769076012.069040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076012/real 1769076012] req@ffff969e0a7fe300 x1855007923683968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076012.069044:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076012.069056:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076012.069074:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076012.069076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923683968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076012.069095:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923684480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076012.069096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076012.069111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076012.069150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923683968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076012.069205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076012.069230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076012.069242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076012/real 1769076012] req@ffff969e0a7fed80 x1855007923684096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076012.069254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923684096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076012.069263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076012.069267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076012.069270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076012.069276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076012.069290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076012.069298:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076012/real 1769076012] req@ffff969e0a7fe680 x1855007923684224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076012.069325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923684224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076012.069333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076012.069336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076012.069339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076012.069344:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076012.069362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076012.069371:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076012/real 1769076012] req@ffff969e0a7fc700 x1855007923684352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076012.069382:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923684352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076012.069389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076012.069392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076012.069395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076012.069400:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076012.069413:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076012.069420:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076012/real 1769076012] req@ffff969e0a7fd880 x1855007923684480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076012.069430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923684480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076012.069450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076012.069454:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076012.069457:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076012.388770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076013.092782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076013.412732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076014.116814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076014.436679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076015.140770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076015.460755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076016.164759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076016.484748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076017.188761:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076017.188778:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076017.188792:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769076017.188805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076017.188810:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2706 00000100:00080000:3.0:1769076017.188831:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076017.188843:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076017.188905:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923684608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076017.188945:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076017.188950:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076017.188952:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076017.188958:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2706 00000100:00080000:3.0:1769076017.188962:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076017.188967:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076017.188981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076017.189002:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923684736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076017.189015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923684608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076017.189023:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076017.189079:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076017.189083:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076017.189093:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2706 00000100:00080000:3.0:1769076017.189128:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076017.189135:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076017.189145:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076017.189165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076017.189173:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923684864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076017.189193:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076017.189199:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076017.189201:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076017.189209:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2706 00000100:00080000:3.0:1769076017.189223:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076017.189230:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076017.189237:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076017.189240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076017/real 1769076017] req@ffff969e0a7fea00 x1855007923684608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076017.189262:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923684992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076017.189277:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923684608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076017.189280:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076017.189303:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076017.189313:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076017.189334:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2706 00000100:00080000:1.0:1769076017.189337:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076017.189346:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076017.189350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076017.189372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923684608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076017.189383:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076017.189397:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076017.189411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076017.189428:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923685120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076017.189447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076017.189501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076017/real 1769076017] req@ffff969e0a7fd880 x1855007923684736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076017.189521:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923684736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076017.189538:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076017.189543:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076017.189553:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076017.189565:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076017.189624:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076017.189640:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076017/real 1769076017] req@ffff969e0a7fc700 x1855007923684864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076017.189656:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923684864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076017.189666:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076017.189670:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076017.189674:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076017.189682:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076017.189705:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076017.189716:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076017/real 1769076017] req@ffff969e0a7fe680 x1855007923684992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076017.189730:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923684992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076017.189751:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076017.189756:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076017.189760:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923684992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076017.189786:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076017.189811:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076017.189823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076017/real 1769076017] req@ffff969e0a7fed80 x1855007923685120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076017.189837:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923685120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076017.189848:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076017.189852:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076017.189856:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076017.508769:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076018.212746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076018.532777:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076019.236866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076019.556707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076020.260718:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076020.580724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076021.284709:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076021.604675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076022.308762:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076022.308785:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076022.308793:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769076022.308806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076022.308811:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2711 00000100:00080000:3.0:1769076022.308836:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076022.308853:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076022.308930:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923685248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076022.309013:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076022.309018:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076022.309021:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076022.309028:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2711 00000100:00080000:3.0:1769076022.309035:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076022.309041:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076022.309057:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076022.309079:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923685376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076022.309095:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076022.309099:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076022.309103:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076022.309115:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2711 00000100:00100000:1.0:1769076022.309118:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923685248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076022.309121:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076022.309133:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076022.309173:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076022.309198:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923685504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076022.309214:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076022.309219:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076022.309222:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076022.309228:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2711 00000100:00080000:3.0:1769076022.309234:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076022.309242:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076022.309248:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076022.309264:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923685632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076022.309277:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076022.309282:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076022.309285:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076022.309296:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2711 00000800:00000100:1.0:1769076022.309304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076022.309307:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076022.309320:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076022.309327:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076022.309344:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923685760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076022.309396:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076022/real 1769076022] req@ffff969e0a7fe300 x1855007923685248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076022.309439:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923685248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076022.309462:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076022.309471:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076022.309481:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923685248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076022.309526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076022.309576:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076022.309648:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076022/real 1769076022] req@ffff969e0a7fca80 x1855007923685376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076022.309678:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923685376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076022.309696:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076022.309704:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076022.309711:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076022.309726:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076022.309768:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076022.309791:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076022/real 1769076022] req@ffff969e0a7fd180 x1855007923685504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076022.309844:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923685504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076022.309862:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076022.309869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076022.309876:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076022.309890:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076022.309935:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076022.309993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076022/real 1769076022] req@ffff969e0a7fce00 x1855007923685632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076022.310020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923685632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076022.310038:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076022.310044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076022.310052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076022.310065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076022.310104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076022.310127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076022/real 1769076022] req@ffff969e0a7fdf80 x1855007923685760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076022.310152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923685760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076022.310194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076022.310202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076022.310210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923685760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076022.628716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076023.332757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076023.652722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076024.356803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076024.676710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076025.380709:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076025.700745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076026.404793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076026.724820:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076027.428724:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076027.428743:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1769076027.428783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076027.428783:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076027.428808:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2716 00000100:00080000:3.0:1769076027.428827:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076027.428840:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076027.428935:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923685888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076027.428979:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076027.428985:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076027.428988:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076027.428994:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2716 00000100:00080000:3.0:1769076027.428999:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076027.429006:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076027.429022:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076027.429041:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923686016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076027.429052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923685888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076027.429080:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076027.429087:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076027.429089:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076027.429096:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2716 00000100:00080000:3.0:1769076027.429105:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076027.429115:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076027.429125:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076027.429154:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923686144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076027.429171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076027.429212:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076027.429221:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769076027.429229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076027/real 1769076027] req@ffff969e0a7fc380 x1855007923685888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076027.429230:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076027.429243:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2716 00000100:00080000:3.0:1769076027.429262:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076027.429277:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076027.429290:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076027.429293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923685888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076027.429326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076027.429333:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923686272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076027.429343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076027.429366:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923685888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076027.429381:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076027.429389:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076027.429393:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076027.429401:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2716 00000100:00080000:3.0:1769076027.429411:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076027.429420:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076027.429424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686016:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769076027.429436:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076027.429462:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923686400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076027.429478:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076027.429539:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076027/real 1769076027] req@ffff969e0a7fc000 x1855007923686016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076027.429557:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923686016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076027.429570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076027.429575:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076027.429579:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076027.429621:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076027.429650:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076027.429664:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076027/real 1769076027] req@ffff969e0a7fd500 x1855007923686144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076027.429680:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923686144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076027.429690:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076027.429695:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076027.429699:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076027.429708:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076027.429728:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076027.429739:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076027/real 1769076027] req@ffff969e0a7fdc00 x1855007923686272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076027.429753:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923686272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076027.429779:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076027.429786:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076027.429793:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076027.429828:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076027.429902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076027.429924:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076027/real 1769076027] req@ffff969e0a7ff800 x1855007923686400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076027.429940:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923686400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076027.429951:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076027.429956:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076027.429960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076027.748698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076028.452733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076028.772720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076029.476796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076029.796745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076030.500730:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076030.820732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076031.524781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076031.844725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076032.548843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076032.548848:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076032.548864:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076032.548872:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076032.548890:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2721 00000100:00080000:0.0:1769076032.548906:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076032.548918:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076032.548997:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923686528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076032.549049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076032.549057:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076032.549061:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076032.549073:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2721 00000100:00080000:0.0:1769076032.549082:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076032.549092:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076032.549099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923686528:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076032.549112:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076032.549147:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923686656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076032.549208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076032.549215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076032.549219:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076032.549228:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2721 00000100:00080000:0.0:1769076032.549238:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076032.549247:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076032.549259:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076032.549290:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923686784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076032.549312:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686656:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769076032.549317:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076032.549332:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076032.549336:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076032.549347:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2721 00000100:00080000:0.0:1769076032.549393:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076032.549405:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076032.549418:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076032.549431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076032.549448:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923686912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076032.549477:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076032.549485:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076032.549490:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076032.549500:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2721 00000100:00100000:1.0:1769076032.549510:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686912:192.168.204.143@tcp:8:kworker.0 00000100:00080000:0.0:1769076032.549531:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076032.549546:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076032.549559:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076032.549647:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923687040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076032.549764:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076032.868714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076033.572806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076033.892724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076034.596869:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076034.916792:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076035.620914:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076035.620939:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:1.0:1769076035.621021:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076035.621066:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076035.621117:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076035.621126:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076035.621135:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076035.621144:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769076035.621195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076032/real 1769076035] req@ffff969e06a60380 x1855007923687040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076035.621239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923687040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076035.621252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076035.621258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076035.621262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076035.621271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076032/real 1769076035] req@ffff969e06a61180 x1855007923686912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076035.621287:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923686912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076035.621295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076035.621298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076035.621303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076035.621310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076032/real 1769076035] req@ffff969e06a61500 x1855007923686784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076035.621323:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923686784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076035.621332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076035.621360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076035.621364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076035.621372:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076032/real 1769076035] req@ffff969e06a62680 x1855007923686656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076035.621389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923686656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076035.621398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076035.621402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076035.621405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923686656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076035.621413:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076032/real 1769076035] req@ffff969e06a61f80 x1855007923686528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076035.621427:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923686528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076035.621434:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076035.621437:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076035.621440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923686528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076035.940816:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076036.644871:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076036.964689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076037.668837:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076037.668864:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076037.668870:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076037.668882:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2726 00000100:00080000:1.0:1769076037.668895:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076037.668904:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076037.668953:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923687168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076037.669033:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076037.669040:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076037.669045:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076037.669055:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2726 00000100:00080000:1.0:1769076037.669067:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076037.669076:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076037.669094:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076037.669116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923687168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076037.669132:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923687296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076037.669160:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076037.669165:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076037.669168:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076037.669174:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2726 00000100:00080000:1.0:1769076037.669181:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076037.669188:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076037.669200:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076037.669224:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923687424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076037.669245:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769076037.669251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769076037.669264:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076037.669299:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076037.669313:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2726 00000100:00100000:2.0:1769076037.669316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076037/real 1769076037] req@ffff969e10b60380 x1855007923687168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076037.669355:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076037.669371:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076037.669384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923687168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076037.669385:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076037.669424:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923687552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076037.669489:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076037.669496:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076037.669509:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076037.669511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:1.0:1769076037.669518:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076037.669533:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923687168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769076037.669546:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2726 00000100:00080000:1.0:1769076037.669565:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076037.669578:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076037.669655:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687296:192.168.204.143@tcp:38:kworker.0 00000008:00100000:1.0:1769076037.669680:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076037.669751:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923687680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076037.669755:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076037.669798:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076037/real 1769076037] req@ffff969e10b61180 x1855007923687296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076037.669833:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923687296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076037.669858:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076037.669869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076037.669878:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076037.669896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076037.669944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076037.669972:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076037/real 1769076037] req@ffff969e10b60000 x1855007923687424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076037.669998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923687424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076037.670030:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076037.670044:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076037.670054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687424:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769076037.670076:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076037.670094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076037.670147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076037.670170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076037/real 1769076037] req@ffff969e10b62d80 x1855007923687552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076037.670194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923687552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076037.670211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076037.670218:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076037.670225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076037.670264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076037.670307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076037.670331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076037/real 1769076037] req@ffff969e10b60700 x1855007923687680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076037.670356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923687680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076037.670373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076037.670381:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076037.670388:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076037.988824:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076038.692988:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076039.012798:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076039.716954:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076040.036762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076040.740868:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076041.060745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076041.765227:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076042.084765:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076042.789098:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076042.789113:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076042.789119:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769076042.789149:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076042.789158:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2731 00000100:00080000:2.0:1769076042.789173:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076042.789184:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076042.789267:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923687808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076042.789312:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076042.789318:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076042.789321:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076042.789328:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2731 00000100:00080000:2.0:1769076042.789335:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076042.789342:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076042.789356:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076042.789370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923687808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076042.789381:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923687936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076042.789424:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076042.789430:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076042.789433:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076042.789440:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2731 00000100:00080000:2.0:1769076042.789449:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076042.789456:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076042.789462:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769076042.789467:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076042.789487:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923688064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076042.789513:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076042.789518:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769076042.789523:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076042/real 1769076042] req@ffff969e0a46ea00 x1855007923687808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076042.789523:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076042.789540:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2731 00000100:00080000:2.0:1769076042.789549:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076042.789558:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076042.789567:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076042.789588:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923688192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076042.789589:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923687808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076042.789597:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076042.789710:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076042.789719:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076042.789728:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076042.789731:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2731 00000100:00080000:2.0:1769076042.789742:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076042.789743:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076042.789752:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076042.789757:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923687808:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076042.789765:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076042.789789:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923688320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076042.789797:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076042.789879:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076042.789904:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076042/real 1769076042] req@ffff969e0a46df80 x1855007923687936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076042.789925:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923687936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076042.789939:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076042.789944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076042.789948:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923687936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076042.789955:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076042.789999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076042.790015:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076042/real 1769076042] req@ffff969e0a46f480 x1855007923688064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076042.790030:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923688064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076042.790057:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076042.790076:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076042.790081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076042.790100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076042.790129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076042.790143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076042/real 1769076042] req@ffff969e0a46d500 x1855007923688192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076042.790158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923688192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076042.790173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076042.790180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076042.790212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076042.790241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076042.790272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076042.790285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076042/real 1769076042] req@ffff969e0a46c380 x1855007923688320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076042.790318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923688320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076042.790329:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076042.790334:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076042.790338:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076043.108777:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076043.813053:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076044.132770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076044.836906:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076045.156730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076045.860875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076046.180709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076046.884858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076047.204742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076047.908759:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076047.908773:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076047.908781:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076047.908804:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2737 00000100:00080000:2.0:1769076047.908844:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076047.908858:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076047.908915:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923688448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076047.908919:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076047.908967:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076047.908972:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076047.908975:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076047.908981:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2737 00000100:00080000:2.0:1769076047.908987:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076047.908993:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076047.909006:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076047.909027:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923688576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076047.909043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923688448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076047.909048:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076047.909061:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076047.909064:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076047.909071:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2737 00000100:00080000:2.0:1769076047.909076:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076047.909082:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076047.909088:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076047.909118:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923688704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076047.909131:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076047.909161:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076047.909169:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076047.909172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076047.909181:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2737 00000100:00080000:2.0:1769076047.909191:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076047.909198:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076047.909207:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076047.909227:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923688832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076047.909242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076047/real 1769076047] req@ffff969e0a46f100 x1855007923688448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076047.909245:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076047.909261:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076047.909269:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076047.909286:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2737 00000100:00080000:2.0:1769076047.909293:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076047.909300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923688448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076047.909303:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076047.909317:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076047.909334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076047.909343:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923688960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076047.909349:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076047.909364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923688448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076047.909405:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076047.909438:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076047.909454:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076047/real 1769076047] req@ffff969e0a46f800 x1855007923688576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076047.909470:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923688576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076047.909481:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076047.909485:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076047.909490:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076047.909497:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076047.909570:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076047.909589:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076047/real 1769076047] req@ffff969e0a46c380 x1855007923688704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076047.909674:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923688704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076047.909692:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076047.909699:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076047.909706:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076047.909717:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076047.909750:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076047.909765:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076047/real 1769076047] req@ffff969e0a46d500 x1855007923688832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076047.909780:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923688832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076047.909800:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076047.909804:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076047.909808:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076047.909815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076047.909833:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076047.909843:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076047/real 1769076047] req@ffff969e0a46f480 x1855007923688960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076047.909857:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923688960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076047.909885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076047.909890:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076047.909894:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923688960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076048.228659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076048.932877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076049.252728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076049.957290:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076050.276712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076050.980896:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076051.300711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076052.005133:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076052.324679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076053.029253:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076053.029411:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076053.029426:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076053.029461:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076053.029477:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2742 00000100:00080000:3.0:1769076053.029515:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076053.029527:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076053.029581:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923689088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076053.029694:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076053.029702:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076053.029706:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076053.029715:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2742 00000100:00080000:3.0:1769076053.029723:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076053.029732:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076053.029750:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076053.029783:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923689216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076053.029806:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076053.029812:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076053.029817:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076053.029828:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2742 00000100:00100000:1.0:1769076053.029841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923689088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076053.029843:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076053.029864:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076053.029879:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076053.029911:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923689344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076053.029933:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076053.029954:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076053.029959:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076053.029968:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2742 00000100:00080000:3.0:1769076053.029979:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076053.029985:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076053.029993:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076053.030013:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923689472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076053.030026:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076053.030031:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076053.030034:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076053.030049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076053.030051:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2742 00000100:00080000:3.0:1769076053.030064:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076053.030071:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076053.030078:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076053.030093:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923689600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076053.030174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076053/real 1769076053] req@ffff969e0a7ffb80 x1855007923689088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076053.030219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923689088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076053.030243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076053.030252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076053.030262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923689088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076053.030315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076053.030356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076053.030392:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076053/real 1769076053] req@ffff969e0a7ff800 x1855007923689216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076053.030453:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923689216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076053.030498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076053.030505:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076053.030510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076053.030521:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076053.030571:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076053.030588:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076053/real 1769076053] req@ffff969e0a7fdc00 x1855007923689344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076053.030654:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923689344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076053.030667:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076053.030675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076053.030681:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076053.030690:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076053.030721:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076053.030735:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076053/real 1769076053] req@ffff969e0a7fd500 x1855007923689472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076053.030750:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923689472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076053.030761:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076053.030764:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076053.030779:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076053.030789:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076053.030816:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076053.030829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076053/real 1769076053] req@ffff969e0a7fc000 x1855007923689600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076053.030844:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923689600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076053.030855:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076053.030859:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076053.030862:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076053.348767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076054.053171:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076054.372716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076055.077360:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076055.396737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076056.101119:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076056.420707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076057.125131:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076057.444722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076058.149086:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076058.149120:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076058.149128:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076058.149147:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2747 00000100:00080000:2.0:1769076058.149164:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076058.149177:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076058.149235:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923689728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076058.149295:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076058.149302:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076058.149306:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076058.149315:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2747 00000100:00080000:2.0:1769076058.149334:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076058.149341:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076058.149378:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076058.149406:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923689856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076058.149425:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076058.149431:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076058.149435:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076058.149441:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2747 00000100:00080000:2.0:1769076058.149448:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076058.149455:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076058.149463:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076058.149496:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923689984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076058.149548:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076058.149553:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076058.149563:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076058.149571:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2747 00000100:00080000:2.0:1769076058.149579:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076058.149586:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076058.149643:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076058.149684:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923690112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076058.149705:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076058.149711:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076058.149715:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076058.149722:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2747 00000100:00080000:2.0:1769076058.149729:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076058.149736:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076058.149745:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076058.149766:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923690240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076058.149866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923689728:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769076058.149971:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00010000:00010000:0.0:1769076058.150024:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076058.150035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076058/real 1769076058] req@ffff969e0a46df80 x1855007923689728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076058.150075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923689728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076058.150096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076058.150105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076058.150130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923689728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076058.150146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076058.150197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076058.150225:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076058/real 1769076058] req@ffff969e0a46ea00 x1855007923689856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076058.150265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923689856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076058.150282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076058.150289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076058.150297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076058.150310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076058.150375:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076058.150394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076058/real 1769076058] req@ffff969e0a46ca80 x1855007923689984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076058.150411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923689984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076058.150426:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076058.150433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076058.150440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923689984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076058.150452:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076058.150494:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076058.150532:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076058/real 1769076058] req@ffff969e0a46d880 x1855007923690112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076058.150558:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923690112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076058.150570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076058.150575:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076058.150579:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076058.150587:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076058.150667:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076058.150686:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076058/real 1769076058] req@ffff969e0a46ed80 x1855007923690240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076058.150702:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923690240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076058.150714:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076058.150718:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076058.150722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076058.468768:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076059.172792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076059.492762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076060.196875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076060.516708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076061.221076:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076061.540743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076062.244995:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076062.564720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076063.268770:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076063.268787:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076063.268794:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076063.268807:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2752 00000100:00080000:2.0:1769076063.268861:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076063.268870:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076063.268949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923690368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076063.269009:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076063.269013:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076063.269015:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076063.269019:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2752 00000100:00080000:2.0:1769076063.269024:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076063.269029:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076063.269041:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076063.269062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923690368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076063.269069:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923690496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076063.269083:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076063.269087:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076063.269089:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076063.269094:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2752 00000100:00080000:2.0:1769076063.269099:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076063.269103:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076063.269109:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076063.269123:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923690624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076063.269134:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076063.269146:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076063.269149:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076063.269154:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2752 00000100:00080000:2.0:1769076063.269160:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076063.269162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076063.269169:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076063.269176:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076063.269193:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923690752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076063.269202:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076063.269205:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076063.269206:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076063.269211:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2752 00000100:00100000:3.0:1769076063.269212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076063/real 1769076063] req@ffff969e0a46e300 x1855007923690368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076063.269215:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076063.269222:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076063.269231:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076063.269242:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923690880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076063.269248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923690368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769076063.269255:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076063.269293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076063.269298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076063.269301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923690368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076063.269325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076063.269351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076063.269365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076063/real 1769076063] req@ffff969e0a46ce00 x1855007923690496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076063.269378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923690496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076063.269387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076063.269391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076063.269395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076063.269402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076063.269418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076063.269426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076063/real 1769076063] req@ffff969e0a46ed80 x1855007923690624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076063.269434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923690624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076063.269440:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076063.269442:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076063.269445:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076063.269455:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076063.269470:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076063.269488:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076063/real 1769076063] req@ffff969e0a46d880 x1855007923690752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076063.269496:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923690752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076063.269508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076063.269510:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076063.269513:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076063.269516:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076063.269529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076063.269535:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076063/real 1769076063] req@ffff969e0a46ca80 x1855007923690880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076063.269542:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923690880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076063.269548:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076063.269550:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076063.269552:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923690880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076063.588711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076064.293284:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076064.612769:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076065.317525:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076065.636736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076066.340893:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076066.660767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076067.364856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076067.684735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076068.388723:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076068.388736:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076068.388742:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076068.388769:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2757 00000100:00080000:2.0:1769076068.388785:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076068.388795:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076068.388838:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076068.388849:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923691008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076068.388911:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076068.388916:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076068.388918:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076068.388924:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2757 00000100:00080000:2.0:1769076068.388929:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076068.388934:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076068.388947:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076068.388965:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923691136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076068.388981:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076068.388986:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076068.388988:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076068.388992:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2757 00000100:00080000:2.0:1769076068.388999:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076068.389003:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076068.389009:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076068.389028:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923691264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076068.389035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923691008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076068.389045:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076068.389058:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076068.389066:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076068.389072:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2757 00000100:00080000:2.0:1769076068.389087:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076068.389095:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076068.389105:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076068.389130:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923691392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076068.389144:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076068.389150:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076068.389153:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076068.389157:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2757 00000100:00080000:2.0:1769076068.389206:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076068.389212:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076068.389220:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:3.0:1769076068.389230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076068.389240:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923691520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076068.389307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076068/real 1769076068] req@ffff969e0a46ea00 x1855007923691008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076068.389360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923691008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076068.389375:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076068.389383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076068.389388:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923691008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076068.389400:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076068.389435:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076068.389451:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076068/real 1769076068] req@ffff969e0a46df80 x1855007923691136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076068.389465:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923691136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076068.389476:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076068.389480:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076068.389484:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076068.389543:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076068.389577:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076068.389598:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076068/real 1769076068] req@ffff969e0a46f480 x1855007923691264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076068.389698:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923691264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076068.389729:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076068.389734:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076068.389739:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076068.389748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076068.389783:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076068.389798:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076068/real 1769076068] req@ffff969e0a46d500 x1855007923691392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076068.389813:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923691392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076068.389825:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076068.389829:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076068.389833:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076068.389840:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076068.389888:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076068.389902:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076068/real 1769076068] req@ffff969e0a46c380 x1855007923691520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076068.389916:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923691520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076068.389927:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076068.389931:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076068.389945:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076068.708726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076069.412847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076069.732703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076070.436865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076070.756767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076071.461241:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076071.780730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076072.484963:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076072.804723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076073.509022:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076073.509460:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076073.509481:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076073.509490:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076073.509512:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2762 00000100:00080000:2.0:1769076073.509533:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076073.509547:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076073.509692:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923691648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076073.509748:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076073.509785:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076073.509791:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076073.509802:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2762 00000100:00080000:2.0:1769076073.509813:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076073.509821:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923691648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076073.509834:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076073.509865:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076073.509935:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923691776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076073.509940:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076073.509989:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076073.509998:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076073.510004:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076073.510009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076073/real 1769076073] req@ffff969e0a46f800 x1855007923691648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076073.510021:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2762 00000100:00080000:2.0:1769076073.510037:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076073.510044:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923691648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076073.510049:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076073.510069:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1769076073.510075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076073.510102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076073.510108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923691648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076073.510130:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923691904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076073.510160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691776:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076073.510176:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076073.510184:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076073.510189:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076073.510198:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2762 00000100:00080000:2.0:1769076073.510207:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076073.510208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076073.510224:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076073.510245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076073/real 1769076073] req@ffff969e0a46f100 x1855007923691776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076073.510252:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076073.510283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923691776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076073.510291:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923692032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076073.510327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076073.510337:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076073.510345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076073.510348:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076073.510354:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076073.510357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691776:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076073.510374:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2762 00000100:00080000:2.0:1769076073.510386:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076073.510393:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076073.510398:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076073.510410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076073.510419:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923692160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076073.510458:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076073.510498:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076073/real 1769076073] req@ffff969e0a46c380 x1855007923691904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076073.510513:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923691904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076073.510523:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076073.510527:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076073.510531:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923691904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076073.510540:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076073.510561:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076073.510572:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076073/real 1769076073] req@ffff969e0a46d500 x1855007923692032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076073.510584:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923692032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076073.510625:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076073.510629:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076073.510633:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076073.510654:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076073.510693:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076073.510704:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076073/real 1769076073] req@ffff969e0a46f480 x1855007923692160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076073.510715:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923692160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076073.510729:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076073.510733:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076073.510736:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076073.828695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076074.533365:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076074.852754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076075.557199:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076075.876717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076076.580868:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076076.900746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076077.604876:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076077.924735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076078.628767:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076078.628784:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076078.628791:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076078.628808:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2767 00000100:00080000:2.0:1769076078.628828:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076078.628841:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076078.628870:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076078.628920:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923692288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076078.628957:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076078.628990:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076078.628994:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076078.629028:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2767 00000100:00080000:2.0:1769076078.629035:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076078.629041:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076078.629058:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076078.629069:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923692288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076078.629088:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923692416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076078.629108:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076078.629113:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076078.629115:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076078.629121:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2767 00000100:00080000:2.0:1769076078.629126:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076078.629131:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076078.629138:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076078.629153:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923692544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076078.629166:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076078.629172:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076078.629174:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076078.629178:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2767 00000100:00080000:2.0:1769076078.629185:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076078.629190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076078.629197:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076078.629218:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076078.629239:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923692672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076078.629256:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076078.629257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076078/real 1769076078] req@ffff969e0a46df80 x1855007923692288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076078.629274:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076078.629284:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076078.629300:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2767 00000100:00080000:2.0:1769076078.629308:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076078.629316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923692288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076078.629322:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076078.629337:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769076078.629347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076078.629359:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923692800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076078.629361:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076078.629380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923692288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076078.629443:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076078.629492:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076078.629514:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076078/real 1769076078] req@ffff969e0a46ea00 x1855007923692416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076078.629532:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923692416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076078.629544:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076078.629549:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076078.629554:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076078.629576:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076078.629678:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076078.629703:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076078/real 1769076078] req@ffff969e0a46ca80 x1855007923692544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076078.629730:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923692544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076078.629748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076078.629755:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076078.629762:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076078.629775:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076078.629817:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076078.629839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076078/real 1769076078] req@ffff969e0a46d880 x1855007923692672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076078.629864:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923692672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076078.629883:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076078.629888:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076078.629892:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076078.629901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076078.629930:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076078.629944:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076078/real 1769076078] req@ffff969e0a46ed80 x1855007923692800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076078.629972:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923692800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076078.629984:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076078.629988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076078.629992:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923692800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076078.948744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076079.652882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076079.972720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076080.676798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076080.996723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076081.700797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076082.020750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076082.724869:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076083.044716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076083.748745:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076083.748759:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076083.748766:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076083.748791:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2772 00000100:00080000:2.0:1769076083.748806:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076083.748815:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076083.748862:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923692928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076083.748882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076083.748938:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076083.748948:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076083.748952:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076083.748960:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2772 00000100:00080000:2.0:1769076083.748992:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076083.749002:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076083.749017:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076083.749023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923692928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076083.749042:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923693056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076083.749062:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076083.749068:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076083.749071:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076083.749077:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2772 00000100:00080000:2.0:1769076083.749113:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076083.749121:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:3.0:1769076083.749125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769076083.749130:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076083.749150:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923693184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076083.749168:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076083.749174:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1769076083.749176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076083/real 1769076083] req@ffff969e0a46ce00 x1855007923692928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076083.749178:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076083.749185:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2772 00000100:00080000:2.0:1769076083.749202:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076083.749208:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076083.749215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923692928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076083.749217:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076083.749233:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923693312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076083.749236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076083.749244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076083.749252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923692928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076083.749273:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076083.749279:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076083.749282:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076083.749288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693056:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076083.749289:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2772 00000100:00080000:2.0:1769076083.749297:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076083.749304:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076083.749313:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:3.0:1769076083.749332:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076083.749341:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923693440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076083.749355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076083/real 1769076083] req@ffff969e0a46e300 x1855007923693056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076083.749379:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923693056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076083.749424:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076083.749432:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076083.749440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076083.749452:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076083.749483:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076083.749504:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076083/real 1769076083] req@ffff969e0a46ed80 x1855007923693184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076083.749519:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923693184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076083.749529:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076083.749533:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076083.749537:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076083.749544:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076083.749563:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076083.749580:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076083/real 1769076083] req@ffff969e0a46d880 x1855007923693312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076083.749638:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923693312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076083.749649:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076083.749653:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076083.749667:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076083.749690:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076083.749718:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076083.749732:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076083/real 1769076083] req@ffff969e0a46ca80 x1855007923693440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076083.749746:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923693440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076083.749757:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076083.749761:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076083.749765:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076084.068659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076084.772852:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076085.092708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076085.796874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076086.116744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076086.820806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076087.140744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076087.844923:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076088.164763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076088.868767:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076088.868791:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076088.868801:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076088.868825:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2778 00000100:00080000:2.0:1769076088.868868:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076088.868882:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076088.868890:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076088.868950:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923693568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076088.868992:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076088.868998:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076088.869001:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076088.869007:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2778 00000100:00080000:2.0:1769076088.869012:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076088.869017:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076088.869055:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076088.869086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923693568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076088.869094:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923693696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076088.869115:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076088.869121:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076088.869124:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076088.869130:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2778 00000100:00080000:2.0:1769076088.869137:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076088.869142:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076088.869150:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076088.869169:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923693824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076088.869191:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076088.869196:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076088.869199:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076088.869210:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2778 00000800:00000100:3.0:1769076088.869210:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076088.869221:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076088.869230:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076088.869239:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076088.869256:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923693952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076088.869271:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076088.869276:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769076088.869282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076088/real 1769076088] req@ffff969e0a46ea00 x1855007923693568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076088.869291:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076088.869308:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2778 00000100:00080000:2.0:1769076088.869319:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076088.869325:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076088.869333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923693568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076088.869336:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769076088.869361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076088.869362:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923694080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076088.869378:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076088.869393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923693568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076088.869458:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076088.869488:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076088.869520:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076088/real 1769076088] req@ffff969e0a46df80 x1855007923693696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076088.869537:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923693696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076088.869549:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076088.869553:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076088.869558:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076088.869565:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076088.869632:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076088.869649:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076088/real 1769076088] req@ffff969e0a46f480 x1855007923693824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076088.869665:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923693824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076088.869676:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076088.869680:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076088.869684:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076088.869692:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076088.869717:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076088.869729:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076088/real 1769076088] req@ffff969e0a46d500 x1855007923693952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076088.869743:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923693952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076088.869764:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076088.869769:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076088.869773:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923693952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076088.869780:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076088.869806:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076088.869819:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076088/real 1769076088] req@ffff969e0a46c380 x1855007923694080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076088.869868:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923694080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076088.869888:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076088.869894:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076088.869901:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076089.188724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076089.892909:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076090.212765:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076090.917096:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076091.236731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076091.941419:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076092.260732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076092.965209:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076093.284712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076093.989009:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076093.989029:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076093.989037:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076093.989056:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2783 00000100:00080000:2.0:1769076093.989074:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076093.989087:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076093.989119:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076093.989169:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923694208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076093.989209:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076093.989215:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076093.989218:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076093.989225:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2783 00000100:00080000:2.0:1769076093.989231:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076093.989245:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076093.989257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923694208:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076093.989259:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076093.989300:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923694336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076093.989318:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076093.989322:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076093.989325:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076093.989332:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2783 00000100:00080000:2.0:1769076093.989339:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076093.989347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076093.989349:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076093.989362:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076093.989403:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923694464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076093.989424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076093/real 1769076093] req@ffff969e0a46f100 x1855007923694208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076093.989427:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076093.989460:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076093.989473:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076093.989481:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2783 00000100:00100000:3.0:1769076093.989486:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923694208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076093.989488:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076093.989497:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076093.989509:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769076093.989516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076093.989526:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923694592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076093.989530:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076093.989545:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923694208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076093.989547:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076093.989551:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076093.989553:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076093.989570:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2783 00000100:00080000:2.0:1769076093.989574:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076093.989578:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076093.989584:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076093.989594:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076093.989631:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923694720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076093.989688:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076093.989710:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076093/real 1769076093] req@ffff969e0a46f800 x1855007923694336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076093.989724:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923694336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076093.989732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076093.989736:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076093.989739:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076093.989770:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076093.989794:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076093.989805:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076093/real 1769076093] req@ffff969e0a46c380 x1855007923694464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076093.989816:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923694464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076093.989824:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076093.989827:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076093.989829:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076093.989835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076093.989849:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076093.989857:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076093/real 1769076093] req@ffff969e0a46d500 x1855007923694592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076093.989876:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923694592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076093.989884:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076093.989887:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076093.989890:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076093.989906:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076093.989923:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076093.989932:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076093/real 1769076093] req@ffff969e0a46f480 x1855007923694720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076093.989942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923694720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076093.989949:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076093.989952:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076093.989954:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076094.308727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076095.012997:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076095.332780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076096.037272:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076096.356753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076097.060925:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076097.380811:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076098.084850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076098.404804:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076099.108762:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076099.108776:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076099.108782:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076099.108796:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2788 00000100:00080000:2.0:1769076099.108812:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076099.108825:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076099.108884:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923694848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076099.108886:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076099.108930:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076099.108938:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076099.108943:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076099.108951:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2788 00000100:00080000:2.0:1769076099.108959:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076099.108967:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076099.108981:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076099.109003:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923694976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076099.109020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923694848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076099.109022:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076099.109055:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076099.109060:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076099.109070:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2788 00000100:00080000:2.0:1769076099.109081:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076099.109089:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076099.109100:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076099.109130:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923695104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076099.109153:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076099.109160:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076099.109165:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076099.109176:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2788 00000100:00080000:2.0:1769076099.109185:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076099.109191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694976:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076099.109199:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076099.109214:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076099.109239:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923695232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076099.109256:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076099.109263:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769076099.109265:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695104:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076099.109268:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076099.109285:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2788 00000100:00080000:2.0:1769076099.109294:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076099.109299:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076099.109319:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076099.109326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076099.109351:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923695360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076099.109396:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076099.428759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076100.133089:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076100.452715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076101.156948:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076101.476761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076102.180903:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769076102.180960:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769076102.181008:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076102.181025:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076102.181063:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076102.181081:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076102.181088:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076102.181095:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:3.0:1769076102.181130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076099/real 1769076102] req@ffff969e0a46ed80 x1855007923695360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076102.181162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923695360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076102.181178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076102.181184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076102.181190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076102.181200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076099/real 1769076102] req@ffff969e0a46d880 x1855007923695232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076102.181217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923695232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076102.181228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076102.181233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076102.181238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076102.181246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076099/real 1769076102] req@ffff969e0a46ea00 x1855007923694976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076102.181263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923694976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076102.181280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076102.181285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076102.181290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923694976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076102.181318:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076099/real 1769076102] req@ffff969e0a46ca80 x1855007923695104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076102.181335:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923695104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076102.181344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076102.181348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076102.181352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076102.181360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076099/real 1769076102] req@ffff969e0a46df80 x1855007923694848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076102.181374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923694848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076102.181383:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076102.181387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076102.181390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923694848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076102.500745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076103.204800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076103.524665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076104.228738:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076104.228749:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076104.228754:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076104.228765:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2793 00000100:00080000:2.0:1769076104.228777:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076104.228784:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076104.228823:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923695488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076104.228858:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076104.228863:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076104.228865:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076104.228869:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2793 00000100:00080000:2.0:1769076104.228873:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076104.228876:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076104.228904:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:2.0:1769076104.228905:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076104.228951:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923695616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076104.228970:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076104.228977:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076104.228980:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076104.229003:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2793 00000100:00080000:2.0:1769076104.229007:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076104.229016:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076104.229016:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923695488:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076104.229029:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076104.229045:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923695744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076104.229056:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076104.229060:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076104.229062:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076104.229065:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2793 00000100:00080000:2.0:1769076104.229079:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076104.229084:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:3.0:1769076104.229090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769076104.229092:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076104.229107:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923695872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076104.229118:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076104.229121:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769076104.229129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076104/real 1769076104] req@ffff969e0a46e300 x1855007923695488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076104.229131:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076104.229142:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2793 00000100:00080000:2.0:1769076104.229151:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076104.229157:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076104.229163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923695488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076104.229164:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769076104.229181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076104.229184:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923696000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076104.229212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076104.229219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923695488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076104.229228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076104.229255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076104.229268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076104/real 1769076104] req@ffff969e0a46ce00 x1855007923695616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076104.229283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923695616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076104.229292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076104.229295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076104.229299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076104.229326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076104.229347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076104.229360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076104/real 1769076104] req@ffff969e0a46df80 x1855007923695744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076104.229374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923695744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076104.229384:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076104.229388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076104.229392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076104.229398:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076104.229430:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076104.229444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076104/real 1769076104] req@ffff969e0a46ca80 x1855007923695872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076104.229458:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923695872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076104.229467:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076104.229471:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076104.229475:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923695872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076104.229481:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076104.229501:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076104.229513:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076104/real 1769076104] req@ffff969e0a46ea00 x1855007923696000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076104.229525:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923696000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076104.229534:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076104.229538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076104.229542:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076104.548662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076105.253001:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076105.572696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076106.276796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076106.596717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076107.300877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076107.620707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076108.324847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076108.644685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076109.348755:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076109.348768:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076109.348774:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076109.348790:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2798 00000100:00080000:2.0:1769076109.348805:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076109.348816:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076109.348823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076109.348866:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923696128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076109.348904:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076109.348909:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076109.348911:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076109.348916:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2798 00000100:00080000:2.0:1769076109.348921:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076109.348926:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076109.348939:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076109.348956:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923696256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076109.348995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923696128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076109.349020:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076109.349027:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076109.349030:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076109.349038:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2798 00000100:00080000:2.0:1769076109.349049:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076109.349055:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076109.349062:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076109.349081:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923696384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076109.349092:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076109.349097:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076109.349099:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076109.349104:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2798 00000100:00080000:2.0:1769076109.349119:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076109.349123:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076109.349130:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:3.0:1769076109.349149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076109.349150:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923696512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076109.349172:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076109.349177:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076109.349180:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076109.349187:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2798 00000100:00080000:2.0:1769076109.349204:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076109.349210:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076109.349218:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076109.349239:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923696640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076109.349247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076109/real 1769076109] req@ffff969e0a46d880 x1855007923696128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076109.349350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923696128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076109.349366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076109.349374:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076109.349380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923696128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076109.349424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076109.349461:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076109.349508:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076109/real 1769076109] req@ffff969e0a46ed80 x1855007923696256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076109.349538:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923696256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076109.349556:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076109.349564:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076109.349571:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076109.349586:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076109.349669:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076109.349686:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076109/real 1769076109] req@ffff969e0a46f480 x1855007923696384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076109.349704:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923696384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076109.349733:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076109.349738:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076109.349743:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076109.349752:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076109.349796:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076109.349812:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076109/real 1769076109] req@ffff969e0a46d500 x1855007923696512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076109.349828:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923696512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076109.349839:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076109.349844:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076109.349848:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076109.349856:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076109.349877:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076109.349890:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076109/real 1769076109] req@ffff969e0a46c380 x1855007923696640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076109.349904:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923696640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076109.349915:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076109.349918:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076109.349932:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076109.668724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076110.372798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076110.692708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076111.396818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076111.716690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076112.420881:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076112.740690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076113.444797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076113.764729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076114.468746:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076114.468760:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076114.468767:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076114.468779:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2803 00000100:00080000:2.0:1769076114.468794:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076114.468804:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076114.468852:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923696768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076114.468892:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076114.468898:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076114.468901:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076114.468906:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2803 00000100:00080000:2.0:1769076114.468913:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769076114.468914:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076114.468921:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076114.468936:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076114.468953:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923696896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076114.468972:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076114.468989:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923696768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076114.469009:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076114.469013:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076114.469019:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2803 00000100:00080000:2.0:1769076114.469024:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076114.469030:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076114.469038:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076114.469059:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923697024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076114.469075:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076114.469080:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076114.469082:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076114.469087:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2803 00000100:00080000:2.0:1769076114.469119:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076114.469125:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076114.469133:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:3.0:1769076114.469143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076114.469159:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923697152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076114.469175:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076114.469180:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076114.469183:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076114.469191:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2803 00000100:00080000:2.0:1769076114.469202:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076114.469207:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076114.469217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076114/real 1769076114] req@ffff969e0a46f800 x1855007923696768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076114.469225:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076114.469257:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923697280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076114.469272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923696768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076114.469313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076114.469324:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076114.469336:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923696768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076114.469379:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076114.469464:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076114.469491:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076114/real 1769076114] req@ffff969e0a46f100 x1855007923696896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076114.469524:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923696896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076114.469537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076114.469542:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076114.469547:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923696896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076114.469556:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076114.469585:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076114.469669:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076114/real 1769076114] req@ffff969e0a46c380 x1855007923697024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076114.469711:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923697024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076114.469732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076114.469740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076114.469748:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076114.469762:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076114.469803:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076114.469821:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076114/real 1769076114] req@ffff969e0a46d500 x1855007923697152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076114.469837:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923697152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076114.469849:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076114.469853:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076114.469858:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076114.469866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076114.469886:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076114.469898:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076114/real 1769076114] req@ffff969e0a46f480 x1855007923697280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076114.469912:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923697280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076114.469934:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076114.469939:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076114.469944:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076114.788725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076115.492892:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076115.812701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076116.516820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076116.836755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076117.540777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076117.860727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076118.564840:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076118.884735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076119.588856:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1769076119.588872:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076119.588875:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076119.588885:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076119.588906:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2808 00000100:00080000:3.0:1769076119.588925:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076119.588937:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076119.589001:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923697408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076119.589054:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076119.589062:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076119.589066:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076119.589076:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2808 00000100:00080000:3.0:1769076119.589085:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076119.589095:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076119.589115:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076119.589124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923697408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076119.589151:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923697536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076119.589203:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076119.589207:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076119.589210:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076119.589217:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2808 00000100:00080000:3.0:1769076119.589227:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076119.589233:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076119.589241:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076119.589247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076119.589269:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923697664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076119.589286:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076119.589292:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076119.589295:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076119.589302:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2808 00000100:00080000:3.0:1769076119.589308:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076119.589341:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076119.589348:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076119.589368:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076119/real 1769076119] req@ffff969e0a7fc380 x1855007923697408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076119.589369:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923697792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076119.589416:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076119.589426:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076119.589430:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076119.589437:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2808 00000100:00080000:3.0:1769076119.589453:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076119.589454:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923697408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076119.589468:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076119.589485:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076119.589492:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076119.589506:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076119.589509:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923697920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076119.589519:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923697408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076119.589629:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076119.589670:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076119.589690:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076119/real 1769076119] req@ffff969e0a7fdf80 x1855007923697536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076119.589707:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923697536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076119.589719:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076119.589723:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076119.589728:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076119.589736:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076119.589757:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076119.589768:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076119/real 1769076119] req@ffff969e0a7fce00 x1855007923697664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076119.589782:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923697664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076119.589793:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076119.589796:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076119.589800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076119.589807:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076119.589826:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076119.589836:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076119/real 1769076119] req@ffff969e0a7fd180 x1855007923697792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076119.589863:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923697792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076119.589875:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076119.589879:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076119.589883:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076119.589889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076119.589931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076119.589945:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076119/real 1769076119] req@ffff969e0a7fca80 x1855007923697920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076119.589959:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923697920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076119.589970:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076119.589974:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076119.589978:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923697920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076119.908718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076120.612862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076120.932724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076121.636853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076121.956735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076122.660842:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076122.980698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076123.684804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076124.004720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076124.708780:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076124.708799:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076124.708808:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769076124.708824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076124.708827:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2813 00000100:00080000:2.0:1769076124.708848:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076124.708862:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076124.708920:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923698048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076124.708971:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076124.708979:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076124.708983:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076124.708994:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2813 00000100:00080000:2.0:1769076124.709004:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076124.709015:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076124.709032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923698048:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076124.709033:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076124.709075:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923698176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076124.709100:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076124.709108:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076124.709113:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076124.709122:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2813 00000800:00000100:1.0:1769076124.709129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076124.709161:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076124.709176:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076124.709178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076124/real 1769076124] req@ffff969e0a46ed80 x1855007923698048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076124.709191:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076124.709261:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923698304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076124.709276:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923698048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076124.709305:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076124.709307:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076124.709319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076124.709323:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769076124.709329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923698048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076124.709334:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076124.709350:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2813 00000100:00080000:2.0:1769076124.709359:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076124.709372:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698176:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076124.709374:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076124.709393:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076124.709419:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076124.709430:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923698432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076124.709441:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076124/real 1769076124] req@ffff969e0a46d880 x1855007923698176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076124.709488:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076124.709500:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076124.709505:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923698176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076124.709508:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076124.709532:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2813 00000100:00080000:1.0:1769076124.709533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076124.709549:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076124.709550:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076124.709556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698176:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076124.709568:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076124.709573:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076124.709641:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076124.709662:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076124/real 1769076124] req@ffff969e0a46ea00 x1855007923698304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076124.709673:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076124.709685:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923698304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076124.709713:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923698560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076124.709722:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076124.709737:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076124.709746:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076124.709771:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076124.709813:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076124.709834:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076124/real 1769076124] req@ffff969e0a46ca80 x1855007923698432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076124.709851:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923698432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076124.709862:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076124.709868:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076124.709872:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076124.709884:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076124.709902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076124.709913:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076124/real 1769076124] req@ffff969e0a46df80 x1855007923698560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076124.709927:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923698560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076124.709937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076124.709941:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076124.709954:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076125.028733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076125.732816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076126.052712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076126.757720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076127.076707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076127.781782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076128.100672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076128.805081:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076129.124707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076129.829266:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076129.829292:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076129.829301:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076129.829323:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2819 00000100:00080000:2.0:1769076129.829369:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076129.829386:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076129.829393:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076129.829467:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923698688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076129.829539:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076129.829547:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076129.829551:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076129.829560:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2819 00000100:00080000:2.0:1769076129.829568:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076129.829577:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076129.829634:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076129.829665:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923698816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076129.829687:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076129.829694:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076129.829698:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076129.829709:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2819 00000100:00080000:2.0:1769076129.829719:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076129.829738:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076129.829740:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923698688:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076129.829772:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076129.829807:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923698944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076129.829830:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076129.829837:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076129.829841:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076129.829847:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2819 00000100:00080000:2.0:1769076129.829856:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076129.829864:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076129.829875:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076129.829896:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923699072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076129.829912:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076129.829919:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076129.829921:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076129.829928:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2819 00000100:00080000:2.0:1769076129.829934:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076129.829941:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076129.829949:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076129.829966:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923699200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076129.830024:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076129.830090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076129/real 1769076129] req@ffff969e0a46ce00 x1855007923698688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076129.830125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923698688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076129.830166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076129.830175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076129.830181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923698688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076129.830194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076129.830233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076129.830251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076129/real 1769076129] req@ffff969e0a46e300 x1855007923698816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076129.830268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923698816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076129.830279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076129.830284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076129.830308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076129.830344:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076129.830373:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076129.830387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076129/real 1769076129] req@ffff969e0a46df80 x1855007923698944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076129.830402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923698944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076129.830412:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076129.830416:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076129.830421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923698944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076129.830429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076129.830449:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076129.830460:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076129/real 1769076129] req@ffff969e0a46ca80 x1855007923699072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076129.830474:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923699072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076129.830484:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076129.830488:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076129.830492:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076129.830499:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076129.830517:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076129.830547:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076129/real 1769076129] req@ffff969e0a46ea00 x1855007923699200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076129.830572:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923699200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076129.830647:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076129.830661:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076129.830667:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076130.148728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076130.853141:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076131.172725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076131.877188:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076132.196746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076132.900856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076133.220738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076133.924761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076134.244745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076134.948782:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076134.948794:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076134.948800:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076134.948811:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2824 00000100:00080000:3.0:1769076134.948839:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076134.948848:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076134.948897:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923699328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076134.948898:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076134.948942:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076134.948947:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076134.948949:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076134.948954:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2824 00000100:00080000:3.0:1769076134.948958:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076134.948962:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076134.948973:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076134.948988:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923699456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076134.949002:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076134.949011:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923699328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076134.949031:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076134.949034:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076134.949041:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2824 00000100:00080000:3.0:1769076134.949047:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076134.949053:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076134.949088:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076134.949111:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923699584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076134.949162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076134.949164:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076134.949174:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076134.949179:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076134.949190:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2824 00000100:00080000:3.0:1769076134.949201:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076134.949210:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076134.949216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076134/real 1769076134] req@ffff969e0a7fe300 x1855007923699328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076134.949225:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076134.949254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923699328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076134.949266:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923699712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076134.949275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076134.949293:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076134.949304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923699328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076134.949322:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076134.949332:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076134.949338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699456:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076134.949341:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076134.949356:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2824 00000100:00080000:3.0:1769076134.949367:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076134.949379:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076134.949381:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769076134.949404:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076134.949423:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076134/real 1769076134] req@ffff969e0a7fed80 x1855007923699456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076134.949434:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923699840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076134.949467:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923699456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076134.949481:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076134.949486:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076134.949494:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076134.949504:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076134.949530:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076134.949543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076134/real 1769076134] req@ffff969e0a7fe680 x1855007923699584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076134.949558:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923699584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076134.949567:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076134.949571:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076134.949574:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076134.949582:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076134.949650:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076134.949664:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076134/real 1769076134] req@ffff969e0a7fc700 x1855007923699712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076134.949678:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923699712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076134.949696:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076134.949700:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076134.949704:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076134.949724:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076134.949748:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076134.949759:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076134/real 1769076134] req@ffff969e0a7fd880 x1855007923699840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076134.949772:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923699840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076134.949781:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076134.949784:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076134.949787:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923699840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076135.268701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076135.972775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076136.292656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076136.996847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076137.316744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076138.020854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076138.340733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076139.044798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076139.364696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076140.068718:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076140.068731:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076140.068737:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076140.068749:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2829 00000100:00080000:3.0:1769076140.068762:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076140.068771:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076140.068840:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923699968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076140.068875:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076140.068880:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:0.0:1769076140.068881:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076140.068884:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076140.068892:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2829 00000100:00080000:3.0:1769076140.068912:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076140.068917:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076140.068927:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076140.068943:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923700096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076140.068955:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923699968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076140.068958:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076140.068967:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076140.068983:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076140.068988:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2829 00000100:00080000:3.0:1769076140.069028:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076140.069035:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076140.069043:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076140.069061:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923700224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076140.069074:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076140.069078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076140.069080:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076140.069086:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076140.069092:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2829 00000100:00080000:3.0:1769076140.069096:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076140.069118:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076140.069124:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076140.069136:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923700352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076140.069145:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076140.069149:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076140.069151:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076140.069154:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2829 00000100:00080000:3.0:1769076140.069159:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076140.069161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076140/real 1769076140] req@ffff969e0a7fea00 x1855007923699968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076140.069166:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076140.069195:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076140.069213:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923700480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076140.069232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923699968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076140.069285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076140.069296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076140.069320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923699968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076140.069355:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076140.069409:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076140.069426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076140/real 1769076140] req@ffff969e0a7fd880 x1855007923700096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076140.069439:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923700096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076140.069464:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076140.069468:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076140.069471:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076140.069479:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076140.069500:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076140.069510:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076140/real 1769076140] req@ffff969e0a7fc700 x1855007923700224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076140.069522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923700224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076140.069530:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076140.069533:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076140.069536:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076140.069552:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076140.069581:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076140.069598:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076140/real 1769076140] req@ffff969e0a7fe680 x1855007923700352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076140.069617:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923700352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076140.069645:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076140.069650:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076140.069655:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076140.069664:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076140.069752:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076140.069771:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076140/real 1769076140] req@ffff969e0a7fed80 x1855007923700480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076140.069791:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923700480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076140.069806:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076140.069812:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076140.069817:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076140.388683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076141.092791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076141.412741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076142.116714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076142.436814:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076143.140782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076143.460727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076144.164875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076144.484780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076145.188809:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076145.188846:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076145.188853:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076145.188865:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2834 00000100:00080000:0.0:1769076145.188880:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076145.188921:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076145.188942:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076145.188971:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923700608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076145.189022:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076145.189028:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076145.189030:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076145.189036:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2834 00000100:00080000:0.0:1769076145.189041:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076145.189046:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076145.189059:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076145.189085:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923700736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076145.189100:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076145.189105:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076145.189107:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076145.189111:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2834 00000100:00080000:0.0:1769076145.189116:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076145.189123:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076145.189128:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076145.189153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923700608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076145.189159:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923700864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076145.189184:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076145.189188:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076145.189190:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076145.189218:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2834 00000100:00080000:0.0:1769076145.189227:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076145.189232:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076145.189238:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076145.189255:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923700992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076145.189267:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076145.189272:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076145.189274:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076145.189278:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2834 00000100:00080000:0.0:1769076145.189285:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076145.189289:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076145.189296:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076145.189319:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923701120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076145.189368:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076145.189424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076145/real 1769076145] req@ffff969e06a61880 x1855007923700608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076145.189446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923700608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076145.189460:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076145.189465:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076145.189469:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923700608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076145.189503:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076145.189526:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076145.189537:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076145/real 1769076145] req@ffff969e06a63100 x1855007923700736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076145.189549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923700736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076145.189557:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076145.189561:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076145.189564:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076145.189570:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076145.189584:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076145.189612:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076145/real 1769076145] req@ffff969e06a61f80 x1855007923700864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076145.189646:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923700864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076145.189711:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076145.189716:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076145.189721:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076145.189747:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076145.189772:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076145.189783:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076145/real 1769076145] req@ffff969e06a62680 x1855007923700992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076145.189794:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923700992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076145.189803:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076145.189806:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076145.189809:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923700992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076145.189814:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076145.189828:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076145.189836:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076145/real 1769076145] req@ffff969e06a61500 x1855007923701120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076145.189846:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923701120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076145.189869:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076145.189873:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076145.189877:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076145.508779:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076146.212909:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076146.532750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076147.236883:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076147.556718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076148.260883:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076148.580767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076149.284909:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076149.604760:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076150.308846:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076150.308859:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1769076150.308863:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076150.308867:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076150.308882:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2839 00000100:00080000:2.0:1769076150.308898:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076150.308907:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076150.308957:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923701248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076150.309000:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076150.309006:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076150.309008:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076150.309013:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2839 00000100:00080000:2.0:1769076150.309018:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076150.309022:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076150.309033:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076150.309051:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923701376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076150.309065:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076150.309070:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076150.309086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923701248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076150.309101:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076150.309109:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2839 00000100:00080000:2.0:1769076150.309116:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076150.309122:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076150.309129:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076150.309149:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923701504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076150.309163:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076150.309168:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076150.309170:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076150.309175:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2839 00000100:00080000:2.0:1769076150.309192:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076150.309197:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076150.309205:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076150.309211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076150.309227:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923701632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076150.309243:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076150.309248:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076150.309251:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076150.309257:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2839 00000100:00080000:2.0:1769076150.309263:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076150.309270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076150/real 1769076150] req@ffff969e0a46d880 x1855007923701248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076150.309271:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076150.309284:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076150.309319:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923701760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076150.309329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923701248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076150.309359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076150.309369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076150.309379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923701248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076150.309394:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076150.309447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076150.309466:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076150/real 1769076150] req@ffff969e0a46ed80 x1855007923701376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076150.309483:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923701376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076150.309494:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076150.309498:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076150.309503:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076150.309549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076150.309576:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076150.309623:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076150/real 1769076150] req@ffff969e0a46f480 x1855007923701504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076150.309641:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923701504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076150.309651:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076150.309655:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076150.309659:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076150.309667:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076150.309690:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076150.309702:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076150/real 1769076150] req@ffff969e0a46d500 x1855007923701632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076150.309716:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923701632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076150.309726:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076150.309729:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076150.309733:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076150.309740:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076150.309757:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076150.309768:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076150/real 1769076150] req@ffff969e0a46c380 x1855007923701760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076150.309793:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923701760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076150.309832:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076150.309837:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076150.309841:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923701760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076150.628745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076151.332919:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076151.652699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076152.356855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076152.676678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076153.380907:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076153.700725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076154.404815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076154.724676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076155.428859:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076155.428875:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076155.428881:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076155.428894:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2844 00000100:00080000:3.0:1769076155.428920:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076155.428929:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076155.428932:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076155.428988:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923701888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076155.429021:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076155.429025:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076155.429028:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076155.429032:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2844 00000100:00080000:3.0:1769076155.429036:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076155.429042:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076155.429061:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076155.429077:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923701888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076155.429103:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923702016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076155.429117:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076155.429121:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076155.429123:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076155.429128:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2844 00000100:00080000:3.0:1769076155.429148:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076155.429151:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076155.429164:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076155.429174:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923702144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076155.429180:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076155.429189:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076155.429192:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076155.429194:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076155.429198:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2844 00000100:00080000:3.0:1769076155.429203:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076155.429205:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076155.429210:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076155.429218:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076155/real 1769076155] req@ffff969e0a7fe300 x1855007923701888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076155.429222:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923702272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076155.429251:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076155.429254:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076155.429255:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076155.429259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923701888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076155.429271:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2844 00000100:00080000:1.0:1769076155.429275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076155.429277:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076155.429283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076155.429285:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076155.429290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923701888:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076155.429292:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076155.429305:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923702400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076155.429327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076155.429346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076155.429355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076155/real 1769076155] req@ffff969e0a7fca80 x1855007923702016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076155.429364:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923702016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076155.429394:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076155.429397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076155.429399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076155.429404:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076155.429451:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076155.429464:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076155/real 1769076155] req@ffff969e0a7fd180 x1855007923702144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076155.429479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923702144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076155.429488:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076155.429492:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076155.429496:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076155.429503:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076155.429522:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076155.429541:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076155/real 1769076155] req@ffff969e0a7fce00 x1855007923702272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076155.429555:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923702272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076155.429565:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076155.429568:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076155.429581:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076155.429663:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076155.429735:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076155.429784:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076155/real 1769076155] req@ffff969e0a7fdf80 x1855007923702400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076155.429795:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923702400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076155.429802:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076155.429804:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076155.429806:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076155.748828:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076156.452912:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076156.772752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076157.476860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076157.796757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076158.500850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076158.820688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076159.524821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076159.844783:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076160.548776:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076160.548788:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076160.548793:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769076160.548804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076160.548806:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2849 00000100:00080000:0.0:1769076160.548823:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076160.548832:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076160.548874:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923702528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076160.548908:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076160.548912:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076160.548914:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076160.548920:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2849 00000100:00080000:0.0:1769076160.548924:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076160.548929:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076160.548939:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076160.548952:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923702656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076160.548965:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076160.548972:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076160.548975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923702528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076160.548976:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076160.548988:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2849 00000100:00080000:0.0:1769076160.548992:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076160.548996:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076160.549002:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076160.549036:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923702784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076160.549048:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076160.549051:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076160.549053:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076160.549057:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2849 00000800:00000100:1.0:1769076160.549064:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076160.549065:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076160.549075:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076160.549083:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076160.549096:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923702912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076160.549106:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076160.549113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076160/real 1769076160] req@ffff969e06a61180 x1855007923702528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076160.549114:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076160.549119:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076160.549131:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2849 00000100:00080000:0.0:1769076160.549139:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076160.549145:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076160.549151:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076160.549157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923702528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076160.549166:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923703040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076160.549186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076160.549196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076160.549203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923702528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076160.549236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076160.549273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076160.549310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076160/real 1769076160] req@ffff969e06a60380 x1855007923702656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076160.549334:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923702656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076160.549349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076160.549353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076160.549357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076160.549364:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076160.549390:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076160.549401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076160/real 1769076160] req@ffff969e06a62d80 x1855007923702784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076160.549414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923702784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076160.549423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076160.549426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076160.549430:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076160.549436:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076160.549452:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076160.549460:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076160/real 1769076160] req@ffff969e06a60a80 x1855007923702912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076160.549472:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923702912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076160.549492:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076160.549496:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076160.549500:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923702912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076160.549506:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076160.549527:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076160.549537:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076160/real 1769076160] req@ffff969e06a63800 x1855007923703040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076160.549549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923703040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076160.549557:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076160.549560:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076160.549563:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076160.868717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076161.572835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076161.892733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076162.596810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076162.916709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076163.620891:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076163.940704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076164.644927:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076164.964847:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076165.668747:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076165.668766:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076165.668774:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769076165.668864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076165.669060:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2854 00000100:00080000:0.0:1769076165.669080:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076165.669091:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076165.669144:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923703168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076165.669208:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076165.669214:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076165.669219:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076165.669225:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2854 00000100:00080000:0.0:1769076165.669230:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076165.669236:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076165.669250:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076165.669277:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923703296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076165.669289:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923703168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076165.669328:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076165.669337:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076165.669360:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076165.669373:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2854 00000100:00080000:0.0:1769076165.669405:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076165.669413:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076165.669423:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076165.669450:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923703424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076165.669484:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703296:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769076165.669530:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076165.669537:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076165.669572:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076165.669579:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2854 00000100:00080000:0.0:1769076165.669586:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076165.669659:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076165.669674:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076165.669712:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076165.669718:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923703552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076165.669751:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076165.669757:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076165.669761:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076165.669772:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2854 00000100:00100000:1.0:1769076165.669782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703552:192.168.204.143@tcp:8:kworker.0 00000100:00080000:0.0:1769076165.669791:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076165.669802:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076165.669815:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076165.669840:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923703680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076165.669901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076165.988826:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076166.693078:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076167.012765:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076167.716848:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076168.036708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076168.740821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769076168.741149:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769076168.741246:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076168.741300:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076168.741399:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076168.741418:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076168.741438:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076168.741461:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076168.741552:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076165/real 1769076168] req@ffff969e06a60a80 x1855007923703552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076168.741708:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923703552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076168.741734:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076168.741747:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076168.741756:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076168.741772:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076165/real 1769076168] req@ffff969e06a63800 x1855007923703424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076168.741795:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923703424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076168.741811:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076168.741817:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076168.741823:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076168.741836:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076165/real 1769076168] req@ffff969e06a60700 x1855007923703296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076168.741858:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923703296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076168.741886:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076168.741894:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076168.741900:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076168.741914:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076165/real 1769076168] req@ffff969e06a60000 x1855007923703168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076168.741953:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923703168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076168.741968:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076168.741973:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076168.741979:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923703168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076168.742028:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076165/real 1769076168] req@ffff969e06a62d80 x1855007923703680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076168.742052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923703680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076168.742066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076168.742080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076168.742087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076169.060783:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076169.764860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076170.084686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076170.788762:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076170.788781:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076170.788800:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076170.788817:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2859 00000100:00080000:0.0:1769076170.788883:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076170.788961:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076170.788990:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076170.789032:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923703808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076170.789107:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076170.789116:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076170.789120:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076170.789130:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2859 00000100:00080000:0.0:1769076170.789139:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076170.789148:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076170.789169:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076170.789197:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923703936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076170.789216:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076170.789221:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076170.789225:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076170.789238:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923703808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076170.789239:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2859 00000100:00080000:0.0:1769076170.789252:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076170.789263:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076170.789271:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076170.789291:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923704064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076170.789304:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076170.789320:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076170.789324:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076170.789330:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2859 00000100:00080000:0.0:1769076170.789340:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076170.789348:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076170.789355:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076170.789375:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923704192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076170.789378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076170.789404:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076170.789410:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076170.789412:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076170.789422:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2859 00000100:00080000:0.0:1769076170.789432:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076170.789439:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076170.789450:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076170.789510:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923704320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076170.789528:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076170/real 1769076170] req@ffff969e06a60380 x1855007923703808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076170.789571:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923703808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076170.789635:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076170.789645:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076170.789653:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923703808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076170.789725:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076170.789774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076170.789799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076170/real 1769076170] req@ffff969e06a61180 x1855007923703936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076170.789822:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923703936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076170.789838:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076170.789843:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076170.789849:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923703936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076170.789862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076170.789900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076170.789921:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076170/real 1769076170] req@ffff969e06a61500 x1855007923704064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076170.789942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923704064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076170.789956:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076170.789962:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076170.789968:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076170.789980:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076170.790017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076170.790036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076170/real 1769076170] req@ffff969e06a62680 x1855007923704192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076170.790073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923704192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076170.790088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076170.790094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076170.790099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076170.790111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076170.790149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076170.790171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076170/real 1769076170] req@ffff969e06a61f80 x1855007923704320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076170.790190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923704320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076170.790204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076170.790210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076170.790215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076171.108725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076171.812764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076172.132651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076172.836782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076173.156730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076173.860716:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076174.180746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076174.884750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076175.204678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076175.908821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076175.908824:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076175.908839:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076175.908846:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076175.908862:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2865 00000100:00080000:3.0:1769076175.908879:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076175.908890:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076175.908946:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923704448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076175.909007:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076175.909012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076175.909015:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076175.909021:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2865 00000100:00080000:3.0:1769076175.909027:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076175.909034:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076175.909048:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076175.909068:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923704576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076175.909084:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076175.909090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923704448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076175.909090:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076175.909096:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076175.909110:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2865 00000100:00080000:3.0:1769076175.909116:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076175.909120:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076175.909136:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076175.909168:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923704704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076175.909182:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076175.909187:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076175.909190:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076175.909195:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2865 00000100:00080000:3.0:1769076175.909212:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076175.909218:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076175.909227:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076175.909231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076175.909263:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923704832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076175.909278:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076175.909283:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076175.909286:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076175.909291:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2865 00000100:00080000:3.0:1769076175.909297:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076175.909302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076175/real 1769076175] req@ffff969e0a7fc380 x1855007923704448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076175.909309:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076175.909324:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076175.909345:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923704960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076175.909349:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923704448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076175.909437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076175.909446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076175.909453:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923704448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076175.909490:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076175.909526:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076175.909543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076175/real 1769076175] req@ffff969e0a7fc000 x1855007923704576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076175.909560:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923704576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076175.909571:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076175.909575:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076175.909579:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076175.909586:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076175.909643:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076175.909657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076175/real 1769076175] req@ffff969e0a7fd500 x1855007923704704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076175.909672:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923704704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076175.909683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076175.909687:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076175.909691:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076175.909712:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076175.909740:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076175.909753:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076175/real 1769076175] req@ffff969e0a7fdc00 x1855007923704832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076175.909767:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923704832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076175.909778:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076175.909781:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076175.909785:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076175.909792:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076175.909810:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076175.909821:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076175/real 1769076175] req@ffff969e0a7ff800 x1855007923704960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076175.909835:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923704960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076175.909845:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076175.909848:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076175.909852:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923704960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076176.228717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076176.932865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076177.252708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076177.956831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076178.276723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076178.980913:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076179.300757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076180.004820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076180.324733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076181.028746:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076181.028759:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076181.028765:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076181.028780:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2870 00000100:00080000:3.0:1769076181.028795:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076181.028805:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076181.028855:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923705088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076181.028898:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1769076181.028925:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076181.028931:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076181.028939:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076181.028950:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2870 00000100:00080000:3.0:1769076181.028960:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076181.028977:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923705088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076181.028977:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076181.029013:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076181.029057:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923705216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076181.029083:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076181.029088:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076181.029090:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076181.029095:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2870 00000100:00080000:3.0:1769076181.029102:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076181.029108:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076181.029119:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076181.029140:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923705344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076181.029161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076181.029167:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076181.029177:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076181.029182:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076181.029195:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2870 00000100:00080000:3.0:1769076181.029208:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076181.029219:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076181.029239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076181/real 1769076181] req@ffff969e0a7ffb80 x1855007923705088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076181.029250:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076181.029305:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923705472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076181.029327:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076181.029331:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076181.029335:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076181.029342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923705088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076181.029352:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2870 00000100:00080000:3.0:1769076181.029373:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076181.029387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076181.029388:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076181.029411:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076181.029412:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076181.029433:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923705088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076181.029448:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923705600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076181.029488:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076181.029529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076181.029547:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076181/real 1769076181] req@ffff969e0a7ff800 x1855007923705216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076181.029564:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923705216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076181.029576:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076181.029580:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076181.029585:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076181.029631:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076181.029657:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076181.029672:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076181/real 1769076181] req@ffff969e0a7fdc00 x1855007923705344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076181.029697:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923705344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076181.029741:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076181.029750:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076181.029758:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076181.029772:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076181.029818:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076181.029836:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076181/real 1769076181] req@ffff969e0a7fd500 x1855007923705472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076181.029852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923705472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076181.029864:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076181.029868:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076181.029873:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076181.029882:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076181.029903:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076181.029915:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076181/real 1769076181] req@ffff969e0a7fc000 x1855007923705600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076181.029930:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923705600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076181.029940:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076181.029944:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076181.029961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076181.348685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076182.052802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076182.372708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076183.076875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076183.396721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076184.100807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076184.420748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076185.124739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076185.444665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076186.148732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076186.148735:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076186.148742:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076186.148747:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076186.148755:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2875 00000100:00080000:2.0:1769076186.148764:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076186.148769:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076186.148827:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923705728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076186.148862:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076186.148867:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076186.148869:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076186.148873:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2875 00000100:00080000:2.0:1769076186.148878:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076186.148883:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076186.148916:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923705728:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076186.148916:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076186.148949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923705856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076186.148958:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076186.148961:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076186.148963:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076186.148966:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2875 00000100:00080000:2.0:1769076186.148969:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076186.148973:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076186.148976:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076186.148986:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923705984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076186.149008:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076186.149013:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076186.149014:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076186.149017:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076186.149021:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2875 00000100:00080000:2.0:1769076186.149025:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076186.149028:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076186.149032:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076186.149041:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923706112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076186.149048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076186/real 1769076186] req@ffff969e0a46f100 x1855007923705728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076186.149050:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076186.149060:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076186.149066:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076186.149073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923705728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076186.149073:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2875 00000100:00080000:1.0:1769076186.149087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076186.149088:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076186.149092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076186.149094:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076186.149099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923705728:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076186.149101:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076186.149112:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923706240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076186.149139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076186.149183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076186.149234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076186/real 1769076186] req@ffff969e0a46f800 x1855007923705856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076186.149263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923705856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076186.149269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076186.149271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076186.149274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076186.149278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076186.149292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076186.149307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076186/real 1769076186] req@ffff969e0a46c380 x1855007923705984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076186.149314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923705984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076186.149320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076186.149322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076186.149324:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923705984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076186.149328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076186.149340:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076186.149346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076186/real 1769076186] req@ffff969e0a46d500 x1855007923706112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076186.149353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923706112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076186.149358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076186.149360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076186.149362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076186.149373:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076186.149383:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076186.149389:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076186/real 1769076186] req@ffff969e0a46f480 x1855007923706240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076186.149396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923706240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076186.149406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076186.149409:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076186.149411:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076186.468676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076187.172817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076187.492716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076188.196805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076188.516701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076189.220788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076189.540736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076190.244801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076190.564690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076191.268699:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076191.268714:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076191.268722:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076191.268748:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2880 00000100:00080000:2.0:1769076191.268764:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076191.268774:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076191.268824:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923706368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076191.268863:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1769076191.268869:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076191.268869:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076191.268874:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076191.268883:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2880 00000100:00080000:2.0:1769076191.268889:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076191.268917:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076191.268928:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923706368:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076191.268931:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076191.268973:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923706496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076191.269013:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076191.269020:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076191.269025:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076191.269037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076191.269043:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2880 00000100:00080000:2.0:1769076191.269056:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076191.269067:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076191.269078:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076191.269101:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923706624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076191.269140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076191/real 1769076191] req@ffff969e0a46ed80 x1855007923706368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076191.269157:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076191.269176:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076191.269184:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076191.269195:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2880 00000100:00100000:1.0:1769076191.269206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923706368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076191.269207:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076191.269227:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076191.269246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076191.269248:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076191.269260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076191.269268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923706368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076191.269277:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923706752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076191.269294:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076191.269308:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076191.269313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706496:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076191.269315:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076191.269332:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2880 00000100:00080000:2.0:1769076191.269341:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076191.269347:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076191.269355:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076191.269370:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076191.269394:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923706880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076191.269398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076191/real 1769076191] req@ffff969e0a46d880 x1855007923706496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076191.269456:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923706496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076191.269467:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076191.269473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076191.269478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076191.269503:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076191.269532:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076191.269564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076191/real 1769076191] req@ffff969e0a46ea00 x1855007923706624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076191.269587:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923706624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076191.269629:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076191.269634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076191.269638:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076191.269646:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076191.269672:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076191.269686:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076191/real 1769076191] req@ffff969e0a46ca80 x1855007923706752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076191.269700:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923706752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076191.269711:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076191.269715:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076191.269718:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076191.269741:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076191.269760:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076191.269771:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076191/real 1769076191] req@ffff969e0a46df80 x1855007923706880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076191.269798:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923706880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076191.269809:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076191.269813:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076191.269817:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923706880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076191.588791:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076192.292857:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076192.612778:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076193.316910:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076193.636738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000008:00020000:0.0:1769076194.340791:0:32796:0:(osc_cache.c:945:osc_extent_wait()) extent ffff969e13116240@{[0 -> 0/1023], [3|0|+|rpc|wiuY|ffff969e02bdb880], [28672|1|+|-|ffff969e067d0800|1024|ffff969e01224f80]} lustre-OST0001-osc-ffff969e12057000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:2.0:1769076194.340980:0:32724:0:(osc_cache.c:945:osc_extent_wait()) extent ffff969e06afd780@{[0 -> 0/1023], [3|0|+|rpc|wihY|ffff969e057fe000], [28672|1|+|-|ffff969e06aae400|1024|ffff969e06b54f80]} lustre-OST0000-osc-ffff969e12057000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:3.0:1769076194.340988:0:32714:0:(osc_cache.c:945:osc_extent_wait()) extent ffff969e06afd480@{[0 -> 0/1023], [3|0|+|rpc|wihY|ffff969e057ff180], [28672|1|+|-|ffff969e06aaec00|1024|ffff969e12718000]} lustre-OST0001-osc-ffff969e12057000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:2.0:1769076194.341020:0:32724:0:(osc_cache.c:945:osc_extent_wait()) ### extent: 000000008f49cd8b ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e06aae400/0x44e2946041b61945 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88de39 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00020000:3.0:1769076194.341022:0:32714:0:(osc_cache.c:945:osc_extent_wait()) ### extent: 00000000fd782c19 ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e06aaec00/0x44e2946041b6195a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x22f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88de71 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000008:00020000:3.0:1769076194.341141:0:32795:0:(osc_cache.c:945:osc_extent_wait()) extent ffff969e13116f00@{[0 -> 0/1023], [3|0|+|rpc|wiuY|ffff969e02bdbc00], [28672|1|+|-|ffff969e067d0000|1024|ffff969e01224f80]} lustre-OST0000-osc-ffff969e12057000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:3.0:1769076194.341159:0:32795:0:(osc_cache.c:945:osc_extent_wait()) ### extent: 000000004abc2fe7 ns: lustre-OST0000-osc-ffff969e12057000 lock: ffff969e067d0000/0x44e2946041b60763 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88ae89 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1769076194.341225:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:0.0:1769076194.347263:0:32796:0:(osc_cache.c:945:osc_extent_wait()) ### extent: 000000003c781924 ns: lustre-OST0001-osc-ffff969e12057000 lock: ffff969e067d0800/0x44e2946041b60778 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0x9b91fc930f88aec1 expref: -99 pid: 31824 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1769076194.660710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076195.365003:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076195.684685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076196.388865:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076196.388885:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1769076196.388890:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076196.388894:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076196.388925:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2885 00000100:00080000:2.0:1769076196.388962:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076196.388975:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076196.389074:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923707008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076196.389177:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076196.389206:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076196.389211:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076196.389221:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2885 00000100:00080000:2.0:1769076196.389231:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076196.389239:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076196.389256:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076196.389285:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923707136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076196.389305:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076196.389313:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076196.389317:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076196.389325:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2885 00000100:00080000:2.0:1769076196.389332:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076196.389365:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076196.389378:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076196.389389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923707008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076196.389411:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923707264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076196.389438:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076196.389445:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076196.389449:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076196.389457:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2885 00000100:00080000:2.0:1769076196.389497:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076196.389509:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076196.389517:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076196.389533:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076196.389566:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923707392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076196.389585:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076196/real 1769076196] req@ffff969e0a46e300 x1855007923707008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076196.389629:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076196.389637:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076196.389641:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076196.389650:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2885 00000100:00080000:2.0:1769076196.389658:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076196.389666:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076196.389680:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076196.389708:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923707520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076196.389728:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923707008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076196.389774:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076196.389791:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076196.389800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923707008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076196.389816:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076196.389877:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076196.389913:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076196/real 1769076196] req@ffff969e0a46ce00 x1855007923707136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076196.389973:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923707136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076196.389991:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076196.390001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076196.390009:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076196.390092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076196.390144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076196.390165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076196/real 1769076196] req@ffff969e0a46df80 x1855007923707264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076196.390181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923707264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076196.390192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076196.390197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076196.390201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076196.390209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076196.390233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076196.390244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076196/real 1769076196] req@ffff969e0a46ca80 x1855007923707392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076196.390258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923707392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076196.390287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076196.390292:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076196.390296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076196.390304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076196.390329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076196.390342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076196/real 1769076196] req@ffff969e0a46ea00 x1855007923707520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076196.390356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923707520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076196.390367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076196.390370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076196.390374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076196.708718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076197.412949:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076197.732696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076198.437194:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076198.756720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076199.460768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076199.780712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076200.484729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076200.804756:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076201.508720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076201.508759:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076201.508776:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076201.508794:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076201.508810:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2890 00000100:00080000:0.0:1769076201.508827:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076201.508838:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076201.508918:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923707648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076201.509035:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076201.509043:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076201.509047:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076201.509056:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2890 00000100:00080000:0.0:1769076201.509064:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076201.509101:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076201.509102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923707648:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076201.509122:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076201.509149:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923707776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076201.509167:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076201.509171:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076201.509174:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076201.509179:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2890 00000100:00080000:0.0:1769076201.509186:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076201.509191:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076201.509196:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076201.509211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076201.509217:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923707904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076201.509232:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076201.509237:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076201.509240:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076201.509246:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2890 00000100:00080000:0.0:1769076201.509253:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076201.509259:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076201.509265:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076201.509271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076201/real 1769076201] req@ffff969e06a63100 x1855007923707648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076201.509283:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923708032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076201.509306:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076201.509315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923707648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076201.509320:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076201.509324:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076201.509333:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2890 00000100:00080000:0.0:1769076201.509343:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076201.509355:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076201.509362:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769076201.509364:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076201.509389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076201.509397:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923708160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076201.509400:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923707648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076201.509449:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076201.509488:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076201.509504:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076201/real 1769076201] req@ffff969e06a61880 x1855007923707776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076201.509520:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923707776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076201.509531:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076201.509536:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076201.509540:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076201.509547:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076201.509566:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076201.509577:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076201/real 1769076201] req@ffff969e06a61f80 x1855007923707904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076201.509621:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923707904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076201.509632:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076201.509636:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076201.509641:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923707904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076201.509660:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076201.509689:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076201.509703:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076201/real 1769076201] req@ffff969e06a62680 x1855007923708032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076201.509718:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923708032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076201.509734:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076201.509741:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076201.509747:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076201.509759:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076201.509796:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076201.509817:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076201/real 1769076201] req@ffff969e06a61500 x1855007923708160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076201.509843:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923708160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076201.509860:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076201.509864:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076201.509868:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076201.828699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076202.532672:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076202.852695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076203.556813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076203.876740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076204.580694:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076204.900716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076205.604711:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076205.924733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076206.628765:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076206.628784:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076206.628796:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769076206.628803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076206.628884:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2895 00000100:00080000:3.0:1769076206.628901:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076206.628913:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076206.628970:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923708288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076206.629011:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076206.629016:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076206.629019:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076206.629025:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2895 00000100:00080000:3.0:1769076206.629031:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076206.629038:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076206.629051:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076206.629078:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923708416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076206.629080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923708288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076206.629128:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076206.629136:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076206.629141:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076206.629150:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2895 00000100:00080000:3.0:1769076206.629160:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076206.629170:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076206.629183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769076206.629183:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076206.629219:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923708544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076206.629235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076206/real 1769076206] req@ffff969e0a7fc380 x1855007923708288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076206.629287:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076206.629289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923708288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076206.629300:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076206.629309:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076206.629325:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076206.629326:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2895 00000100:00080000:1.0:1769076206.629337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076206.629343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923708288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076206.629357:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076206.629368:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076206.629378:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076206.629385:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076206.629403:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923708672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076206.629421:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076206.629425:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076206.629430:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076206.629436:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076206.629442:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2895 00000100:00080000:3.0:1769076206.629449:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076206.629457:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076206.629469:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076206.629471:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076206/real 1769076206] req@ffff969e0a7fdf80 x1855007923708416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076206.629506:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923708800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076206.629507:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923708416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076206.629547:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076206.629560:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076206.629568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076206.629579:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076206.629643:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076206.629660:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076206/real 1769076206] req@ffff969e0a7fce00 x1855007923708544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076206.629720:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923708544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076206.629738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076206.629745:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076206.629753:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076206.629785:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076206.629821:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076206.629837:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076206/real 1769076206] req@ffff969e0a7fd180 x1855007923708672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076206.629884:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923708672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076206.629897:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076206.629901:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076206.629906:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076206.629914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076206.629941:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076206.629956:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076206/real 1769076206] req@ffff969e0a7fca80 x1855007923708800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076206.629970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923708800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076206.629999:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076206.630003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076206.630007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923708800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076206.948701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076207.653001:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076207.972923:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076208.676913:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076208.996762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076209.700856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076210.020716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076210.724838:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076211.044720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076211.748835:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076211.748849:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076211.748856:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769076211.748861:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076211.748880:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2900 00000100:00080000:0.0:1769076211.748896:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076211.748905:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076211.748950:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923708928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076211.748989:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076211.748994:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076211.748997:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076211.749002:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2900 00000100:00080000:0.0:1769076211.749007:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076211.749013:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076211.749026:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076211.749066:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923708928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076211.749094:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923709056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076211.749119:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076211.749126:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076211.749130:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076211.749139:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2900 00000100:00080000:0.0:1769076211.749150:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076211.749159:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076211.749167:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076211.749177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076211.749194:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923709184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076211.749220:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076211.749225:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076211.749228:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076211.749234:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2900 00000100:00080000:0.0:1769076211.749241:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076211.749246:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076211.749251:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076211.749264:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923709312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076211.749272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076211/real 1769076211] req@ffff969e06a61180 x1855007923708928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076211.749289:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076211.749306:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076211.749313:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076211.749329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923708928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076211.749333:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2900 00000100:00080000:0.0:1769076211.749345:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076211.749356:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076211.749360:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769076211.749364:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076211.749372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076211.749391:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923708928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076211.749400:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923709440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076211.749432:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076211.749468:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076211.749485:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076211/real 1769076211] req@ffff969e06a60380 x1855007923709056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076211.749500:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923709056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076211.749511:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076211.749515:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076211.749519:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076211.749541:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076211.749578:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076211.749622:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076211/real 1769076211] req@ffff969e06a62d80 x1855007923709184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076211.749640:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923709184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076211.749650:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076211.749654:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076211.749659:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076211.749666:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076211.749689:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076211.749708:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076211/real 1769076211] req@ffff969e06a60000 x1855007923709312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076211.749721:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923709312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076211.749732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076211.749735:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076211.749739:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076211.749746:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076211.749763:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076211.749813:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076211/real 1769076211] req@ffff969e06a60700 x1855007923709440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076211.749841:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923709440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076211.749852:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076211.749856:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076211.749870:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076212.068720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076212.772907:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076213.092772:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076213.796907:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076214.116841:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076214.820881:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076215.140761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076215.844853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076216.164682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076216.868803:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076216.868816:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076216.868823:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076216.868836:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2906 00010000:00010000:2.0:1769076216.868848:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076216.868872:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076216.868882:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076216.868922:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923709568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076216.868961:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076216.868966:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076216.868969:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076216.868974:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2906 00000100:00080000:3.0:1769076216.868978:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076216.868984:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076216.868996:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076216.869012:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923709696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076216.869026:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923709568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076216.869033:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076216.869039:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076216.869041:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076216.869047:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2906 00000100:00080000:3.0:1769076216.869051:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076216.869056:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076216.869062:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076216.869078:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923709824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076216.869090:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076216.869103:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076216.869106:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076216.869111:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2906 00000100:00080000:3.0:1769076216.869118:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076216.869120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076216.869128:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076216.869138:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076216.869158:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923709952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076216.869172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076216/real 1769076216] req@ffff969e0a7fe300 x1855007923709568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076216.869175:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076216.869188:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076216.869192:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076216.869206:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2906 00000100:00080000:3.0:1769076216.869219:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076216.869233:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076216.869240:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076216.869250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923709568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076216.869258:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923710080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076216.869319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076216.869329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076216.869337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923709568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076216.869373:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076216.869409:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076216.869427:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076216/real 1769076216] req@ffff969e0a7fed80 x1855007923709696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076216.869443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923709696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076216.869486:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076216.869493:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076216.869500:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076216.869512:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076216.869554:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076216.869577:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076216/real 1769076216] req@ffff969e0a7fe680 x1855007923709824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076216.869640:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923709824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076216.869651:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076216.869656:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076216.869660:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076216.869668:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076216.869750:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076216.869772:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076216/real 1769076216] req@ffff969e0a7fc700 x1855007923709952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076216.869798:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923709952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076216.869823:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076216.869830:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076216.869847:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923709952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076216.869856:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076216.869886:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076216.869900:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076216/real 1769076216] req@ffff969e0a7fd880 x1855007923710080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076216.869915:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923710080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076216.869926:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076216.869930:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076216.869933:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076217.188739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076217.892904:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076218.212830:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076218.916877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076219.236725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076219.940774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076220.260726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076220.964792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076221.284711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076221.988683:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076221.988701:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076221.988709:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076221.988725:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2911 00000100:00080000:3.0:1769076221.988743:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076221.988754:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076221.988817:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923710208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076221.988923:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076221.988924:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076221.988931:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076221.988935:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076221.988974:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2911 00000100:00080000:3.0:1769076221.988984:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076221.988996:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076221.989015:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076221.989048:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923710336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076221.989050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923710208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076221.989081:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076221.989087:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076221.989090:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076221.989097:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2911 00000100:00080000:3.0:1769076221.989103:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076221.989108:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076221.989116:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076221.989134:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923710464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076221.989147:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076221.989153:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076221.989156:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076221.989162:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2911 00000100:00080000:3.0:1769076221.989182:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076221.989191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076221.989212:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076221.989223:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076221.989241:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923710592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076221.989255:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076221.989259:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076221.989262:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076221.989268:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2911 00000100:00080000:3.0:1769076221.989274:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076221.989279:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076221.989285:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076221.989294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076221/real 1769076221] req@ffff969e0a7fea00 x1855007923710208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076221.989304:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923710720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076221.989362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923710208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076221.989380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076221.989387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076221.989393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923710208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076221.989467:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076221.989515:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076221.989654:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076221/real 1769076221] req@ffff969e0a7fd880 x1855007923710336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076221.989675:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923710336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076221.989687:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076221.989691:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076221.989695:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076221.989704:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076221.989749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076221.989764:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076221/real 1769076221] req@ffff969e0a7fc700 x1855007923710464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076221.989779:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923710464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076221.989789:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076221.989793:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076221.989797:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076221.989804:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076221.989825:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076221.989836:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076221/real 1769076221] req@ffff969e0a7fe680 x1855007923710592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076221.989849:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923710592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076221.989872:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076221.989877:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076221.989881:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076221.989888:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076221.989913:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076221.989953:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076221/real 1769076221] req@ffff969e0a7fed80 x1855007923710720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076221.989967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923710720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076221.989978:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076221.989982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076221.989985:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076222.308793:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076223.012860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076223.332732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076224.036846:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076224.356704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076225.060864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076225.380740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076226.084739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076226.404772:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076227.108746:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076227.108759:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076227.108766:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076227.108779:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2916 00000100:00080000:0.0:1769076227.108794:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076227.108804:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076227.108811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076227.108853:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923710848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076227.108893:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076227.108898:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076227.108901:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076227.108907:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2916 00000100:00080000:0.0:1769076227.108911:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076227.108917:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076227.108929:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076227.108946:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923710976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076227.108959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923710848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076227.108967:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076227.108991:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076227.108995:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076227.109002:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2916 00000100:00080000:0.0:1769076227.109010:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076227.109016:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076227.109024:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076227.109042:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923711104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076227.109058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076227.109062:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076227.109068:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076227.109071:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076227.109079:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2916 00000100:00080000:0.0:1769076227.109085:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076227.109091:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076227.109100:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076227.109113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076227/real 1769076227] req@ffff969e06a63800 x1855007923710848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076227.109127:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923711232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076227.109149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923710848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076227.109154:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076227.109172:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076227.109181:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076227.109183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076227.109200:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2916 00000100:00080000:1.0:1769076227.109201:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076227.109214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923710848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076227.109218:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076227.109229:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076227.109252:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076227.109269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076227.109276:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923711360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076227.109318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076227.109339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076227/real 1769076227] req@ffff969e06a60a80 x1855007923710976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076227.109356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923710976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076227.109367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076227.109371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076227.109375:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923710976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076227.109383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076227.109401:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076227.109412:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076227/real 1769076227] req@ffff969e06a60700 x1855007923711104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076227.109425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923711104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076227.109436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076227.109439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076227.109443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076227.109449:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076227.109467:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076227.109478:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076227/real 1769076227] req@ffff969e06a60000 x1855007923711232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076227.109503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923711232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076227.109555:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076227.109559:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076227.109563:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076227.109586:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076227.109653:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076227.109677:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076227/real 1769076227] req@ffff969e06a62d80 x1855007923711360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076227.109702:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923711360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076227.109720:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076227.109727:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076227.109734:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076227.428701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076228.132763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076228.452682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076229.156850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076229.476687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076230.180869:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076230.500686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076231.204816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076231.524717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076232.228679:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076232.228687:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076232.228692:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076232.228701:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2921 00000100:00080000:3.0:1769076232.228711:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076232.228718:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076232.228730:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076232.228758:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923711488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076232.228779:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076232.228798:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076232.228800:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076232.228804:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2921 00000100:00080000:3.0:1769076232.228807:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076232.228813:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923711488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076232.228814:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076232.228826:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076232.228839:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923711616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076232.228848:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076232.228850:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076232.228852:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076232.228855:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2921 00000100:00080000:3.0:1769076232.228857:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076232.228859:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076232.228862:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076232.228871:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923711744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076232.228877:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076232.228880:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076232.228882:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076232.228884:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2921 00000100:00080000:3.0:1769076232.228888:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076232.228891:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076232.228899:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076232.228906:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076232.228911:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923711872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076232.228921:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076232.228923:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076232.228925:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076232.228927:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2921 00000100:00080000:3.0:1769076232.228931:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076232.228935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711744:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076232.228937:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076232.228946:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076232.228957:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923712000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076232.228970:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076232.228988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076232.548662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076233.252899:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076233.572723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076234.276914:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076234.596727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076235.300822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769076235.300949:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769076235.300999:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769076235.301024:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076235.301076:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076235.301092:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076235.301106:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076235.301120:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076235.301199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076232/real 1769076235] req@ffff969e0a7fce00 x1855007923711872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076235.301253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923711872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076235.301275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076235.301285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076235.301292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076235.301306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076232/real 1769076235] req@ffff969e0a7fdf80 x1855007923712000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076235.301347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923712000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076235.301435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076235.301444:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076235.301452:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076235.301465:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076232/real 1769076235] req@ffff969e0a7fca80 x1855007923711616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076235.301492:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923711616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076235.301509:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076235.301517:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076235.301524:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076235.301564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076232/real 1769076235] req@ffff969e0a7fd180 x1855007923711744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076235.301630:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923711744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076235.301654:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076235.301664:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076235.301683:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923711744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076235.301700:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076232/real 1769076235] req@ffff969e0a7fe300 x1855007923711488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076235.301727:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923711488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076235.301743:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076235.301749:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076235.301756:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923711488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076235.620690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076236.324883:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076236.644691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076237.348777:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076237.348794:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076237.348801:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076237.348817:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2926 00000100:00080000:3.0:1769076237.348835:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076237.348846:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076237.348860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076237.348904:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923712128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076237.348949:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076237.348955:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076237.348957:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076237.348965:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2926 00000100:00080000:3.0:1769076237.348996:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076237.349000:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923712128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076237.349008:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076237.349034:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076237.349067:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923712256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076237.349083:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076237.349089:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076237.349093:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076237.349098:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2926 00000100:00080000:3.0:1769076237.349103:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076237.349104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076237.349112:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076237.349120:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076237.349138:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923712384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076237.349153:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076237.349158:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076237.349161:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076237.349168:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2926 00000100:00080000:3.0:1769076237.349177:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076237.349180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076237/real 1769076237] req@ffff969e0a7fc380 x1855007923712128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076237.349184:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076237.349196:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076237.349217:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923712512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076237.349232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923712128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076237.349309:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076237.349310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076237.349319:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076237.349324:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076237.349367:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076237.349374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923712128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076237.349409:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712256:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076237.349416:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2926 00000100:00080000:3.0:1769076237.349434:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076237.349446:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076237.349460:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076237.349471:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076237.349498:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923712640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076237.349501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076237/real 1769076237] req@ffff969e0a7fc000 x1855007923712256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076237.349577:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923712256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076237.349628:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076237.349635:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076237.349647:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076237.349658:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076237.349688:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076237.349702:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076237/real 1769076237] req@ffff969e0a7fd500 x1855007923712384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076237.349716:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923712384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076237.349725:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076237.349729:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076237.349733:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076237.349741:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076237.349777:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076237.349790:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076237/real 1769076237] req@ffff969e0a7fdc00 x1855007923712512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076237.349812:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923712512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076237.349821:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076237.349825:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076237.349829:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076237.349852:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076237.349884:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076237.349898:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076237/real 1769076237] req@ffff969e0a7ff800 x1855007923712640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076237.349910:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923712640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076237.349920:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076237.349923:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076237.349927:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076237.668718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076238.372990:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076238.692732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076239.396799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076239.716715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076240.420822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076240.740706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076241.444850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076241.764672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076242.468648:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076242.468657:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076242.468661:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076242.468670:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2931 00000100:00080000:3.0:1769076242.468681:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076242.468689:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076242.468727:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923712768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076242.468730:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076242.468754:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076242.468757:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076242.468759:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076242.468762:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2931 00000100:00080000:3.0:1769076242.468766:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076242.468771:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076242.468786:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923712768:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076242.468789:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076242.468805:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923712896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076242.468814:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076242.468816:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076242.468818:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076242.468821:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2931 00000100:00080000:3.0:1769076242.468825:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076242.468827:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076242.468830:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076242.468838:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923713024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076242.468845:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076242.468847:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076242.468849:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076242.468851:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2931 00000100:00080000:3.0:1769076242.468853:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076242.468857:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076242.468861:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076242.468866:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076242.468875:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923713152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076242.468881:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076242.468884:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076242.468889:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076242.468895:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2931 00000100:00100000:1.0:1769076242.468896:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076242/real 1769076242] req@ffff969e0a7ffb80 x1855007923712768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076242.468898:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076242.468904:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076242.468911:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076242.468920:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923713280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076242.468966:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923712768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076242.468978:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076242.468983:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076242.468988:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923712768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076242.469012:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076242.469032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076242.469045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076242/real 1769076242] req@ffff969e0a7ff800 x1855007923712896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076242.469058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923712896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076242.469067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076242.469070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076242.469074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923712896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076242.469081:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076242.469100:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076242.469108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076242/real 1769076242] req@ffff969e0a7fdc00 x1855007923713024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076242.469116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923713024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076242.469131:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076242.469134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076242.469137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076242.469141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076242.469155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076242.469161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076242/real 1769076242] req@ffff969e0a7fd500 x1855007923713152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076242.469168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923713152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076242.469174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076242.469175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076242.469177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076242.469180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076242.469191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076242.469196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076242/real 1769076242] req@ffff969e0a7fc000 x1855007923713280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076242.469203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923713280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076242.469209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076242.469210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076242.469217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076242.788713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076243.492860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076243.812785:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076244.516743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076244.836669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076245.540784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076245.860701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076246.564906:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076246.884733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076247.588667:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076247.588677:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076247.588681:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076247.588691:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2936 00000100:00080000:3.0:1769076247.588701:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076247.588708:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076247.588743:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923713408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076247.588749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076247.588773:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076247.588776:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076247.588778:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076247.588782:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2936 00000100:00080000:3.0:1769076247.588785:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076247.588788:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076247.588795:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076247.588805:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923713536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076247.588821:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923713408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076247.588838:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076247.588841:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076247.588843:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076247.588847:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2936 00000100:00080000:3.0:1769076247.588851:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076247.588855:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076247.588860:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076247.588870:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923713664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076247.588878:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076247.588881:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076247.588883:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076247.588887:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076247.588891:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2936 00000100:00080000:3.0:1769076247.588896:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076247.588900:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076247.588904:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076247.588916:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923713792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076247.588925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076247/real 1769076247] req@ffff969e0a7fc380 x1855007923713408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076247.588926:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076247.588936:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076247.588939:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076247.588948:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2936 00000100:00080000:3.0:1769076247.588959:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076247.588960:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923713408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076247.588965:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076247.588970:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076247.588981:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923713920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076247.588983:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076247.588995:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076247.589002:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923713408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076247.589024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076247.589051:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076247.589066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076247/real 1769076247] req@ffff969e0a7fe300 x1855007923713536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076247.589083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923713536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076247.589094:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076247.589098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076247.589103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076247.589110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076247.589128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076247.589146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076247/real 1769076247] req@ffff969e0a7fd180 x1855007923713664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076247.589156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923713664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076247.589162:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076247.589182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076247.589185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076247.589191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076247.589209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076247.589217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076247/real 1769076247] req@ffff969e0a7fca80 x1855007923713792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076247.589227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923713792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076247.589234:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076247.589236:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076247.589239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076247.589243:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076247.589254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076247.589260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076247/real 1769076247] req@ffff969e0a7fdf80 x1855007923713920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076247.589268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923713920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076247.589281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076247.589283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076247.589286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923713920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076247.908758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076248.612840:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076248.932750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076249.636834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076249.956718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076250.660865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076250.980688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076251.684847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076252.004799:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076252.708759:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076252.708781:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076252.708789:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769076252.708802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076252.708806:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2941 00000100:00080000:2.0:1769076252.708826:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076252.708837:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076252.708898:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923714048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076252.708985:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076252.708991:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076252.708994:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076252.709000:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2941 00000100:00080000:2.0:1769076252.709006:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076252.709011:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076252.709025:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076252.709043:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923714176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076252.709058:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076252.709063:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076252.709066:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076252.709071:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2941 00000100:00080000:2.0:1769076252.709075:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076252.709125:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076252.709130:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923714048:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076252.709136:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076252.709166:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923714304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076252.709183:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076252.709188:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076252.709191:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076252.709196:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2941 00000100:00080000:2.0:1769076252.709214:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076252.709215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076252.709235:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076252.709258:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076252.709279:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923714432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076252.709296:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076252/real 1769076252] req@ffff969e0a46d880 x1855007923714048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076252.709299:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076252.709323:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076252.709334:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076252.709361:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2941 00000100:00080000:2.0:1769076252.709377:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076252.709395:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076252.709402:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076252.709425:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923714560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076252.709427:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923714048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076252.709477:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076252.709487:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076252.709493:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923714048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076252.709504:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076252.709530:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076252.709542:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076252/real 1769076252] req@ffff969e0a46ed80 x1855007923714176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076252.709552:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923714176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076252.709559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076252.709562:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076252.709565:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076252.709629:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076252.709652:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076252.709666:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076252/real 1769076252] req@ffff969e0a46f480 x1855007923714304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076252.709681:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923714304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076252.709692:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076252.709697:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076252.709701:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076252.709710:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076252.709732:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076252.709742:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076252/real 1769076252] req@ffff969e0a46d500 x1855007923714432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076252.709751:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923714432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076252.709758:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076252.709761:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076252.709763:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076252.709768:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076252.709781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076252.709788:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076252/real 1769076252] req@ffff969e0a46c380 x1855007923714560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076252.709804:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923714560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076252.709811:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076252.709813:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076252.709816:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076253.028704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076253.732775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076254.052731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076254.756801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076255.076750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076255.780881:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076256.100745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076256.804868:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076257.124743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076257.828720:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076257.828738:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076257.828746:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076257.828762:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2947 00000100:00080000:2.0:1769076257.828781:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076257.828793:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076257.828858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076257.828891:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923714688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076257.828950:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076257.828957:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076257.828962:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076257.828970:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2947 00000100:00080000:2.0:1769076257.828978:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076257.828986:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076257.829042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923714688:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076257.829048:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076257.829079:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923714816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076257.829096:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076257.829102:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076257.829105:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076257.829111:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2947 00000100:00080000:2.0:1769076257.829118:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076257.829123:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076257.829129:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076257.829147:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923714944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076257.829164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076257.829167:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076257.829174:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076257.829177:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076257.829196:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2947 00000100:00080000:2.0:1769076257.829204:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076257.829211:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076257.829221:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076257.829226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076257/real 1769076257] req@ffff969e0a46f800 x1855007923714688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076257.829244:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923715072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076257.829261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923714688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076257.829262:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076257.829278:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076257.829284:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076257.829296:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2947 00000100:00080000:1.0:1769076257.829303:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076257.829303:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076257.829314:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076257.829321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769076257.829325:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076257.829333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923714688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076257.829360:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923715200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076257.829379:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076257.829415:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076257.829438:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076257/real 1769076257] req@ffff969e0a46f100 x1855007923714816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076257.829464:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923714816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076257.829528:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076257.829537:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076257.829545:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076257.829559:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076257.829647:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076257.829672:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076257/real 1769076257] req@ffff969e0a46c380 x1855007923714944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076257.829693:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923714944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076257.829705:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076257.829710:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076257.829715:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923714944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076257.829723:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076257.829752:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076257.829765:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076257/real 1769076257] req@ffff969e0a46d500 x1855007923715072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076257.829783:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923715072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076257.829801:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076257.829807:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076257.829828:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076257.829865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076257.829908:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076257.829925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076257/real 1769076257] req@ffff969e0a46f480 x1855007923715200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076257.829941:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923715200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076257.829953:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076257.829957:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076257.829961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076258.148752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076258.852812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076259.172740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076259.876941:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076260.196720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076260.901500:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076261.220692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076261.925179:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076262.244717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076262.948793:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076262.948829:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076262.948838:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769076262.948843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076262.948859:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2952 00000100:00080000:3.0:1769076262.948879:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076262.948892:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076262.948973:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923715328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076262.949013:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076262.949018:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076262.949020:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076262.949026:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2952 00000100:00080000:3.0:1769076262.949030:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076262.949034:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076262.949065:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076262.949085:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923715456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076262.949098:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076262.949129:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076262.949132:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076262.949137:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2952 00000100:00080000:3.0:1769076262.949142:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076262.949148:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076262.949155:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076262.949170:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923715584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076262.949181:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076262.949186:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076262.949188:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076262.949192:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2952 00000100:00080000:3.0:1769076262.949196:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076262.949200:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076262.949206:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076262.949220:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923715712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076262.949231:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076262.949233:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076262.949235:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076262.949238:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2952 00000100:00080000:3.0:1769076262.949241:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076262.949244:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076262.949254:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076262.949264:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923715840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076262.949303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923715328:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769076262.949410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076262.949455:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076262/real 1769076262] req@ffff969e0a7fce00 x1855007923715328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076262.949480:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923715328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076262.949494:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076262.949500:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076262.949506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923715328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076262.949515:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076262.949559:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076262.949575:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076262/real 1769076262] req@ffff969e0a7fed80 x1855007923715456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076262.949624:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923715456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076262.949637:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076262.949642:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076262.949646:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076262.949653:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076262.949678:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076262.949692:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076262/real 1769076262] req@ffff969e0a7fe680 x1855007923715584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076262.949706:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923715584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076262.949716:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076262.949720:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076262.949724:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076262.949732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076262.949755:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076262.949767:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076262/real 1769076262] req@ffff969e0a7fc700 x1855007923715712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076262.949797:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923715712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076262.949808:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076262.949813:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076262.949817:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076262.949825:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076262.949848:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076262.949860:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076262/real 1769076262] req@ffff969e0a7fd880 x1855007923715840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076262.949875:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923715840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076262.949885:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076262.949888:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076262.949892:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923715840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076263.268763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076263.973304:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076264.292680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076264.997019:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076265.316713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076266.020853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076266.340733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076267.044927:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076267.364757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076268.068758:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076268.068774:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076268.068783:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076268.068799:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2957 00000100:00080000:3.0:1769076268.068816:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076268.068870:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076268.068918:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076268.068920:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923715968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076268.068953:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076268.068959:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076268.068962:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076268.068969:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2957 00000100:00080000:3.0:1769076268.068975:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076268.068981:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076268.068994:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076268.069011:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923715968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076268.069013:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923716096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076268.069028:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076268.069033:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076268.069035:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076268.069040:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2957 00000100:00080000:3.0:1769076268.069063:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076268.069068:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076268.069076:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076268.069091:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076268.069091:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923716224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076268.069104:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076268.069109:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076268.069111:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076268.069116:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2957 00000100:00080000:3.0:1769076268.069120:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076268.069125:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076268.069131:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076268.069134:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076268/real 1769076268] req@ffff969e0a7fea00 x1855007923715968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076268.069145:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923716352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076268.069158:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076268.069159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923715968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076268.069163:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076268.069166:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076268.069171:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2957 00000100:00080000:3.0:1769076268.069176:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076268.069181:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076268.069188:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076268.069193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076268.069200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076268.069200:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923716480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076268.069207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923715968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076268.069237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076268.069271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076268.069290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076268/real 1769076268] req@ffff969e0a7fd880 x1855007923716096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076268.069303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923716096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076268.069312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076268.069315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076268.069318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076268.069324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076268.069364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076268.069377:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076268/real 1769076268] req@ffff969e0a7fc700 x1855007923716224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076268.069391:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923716224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076268.069401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076268.069404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076268.069408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076268.069452:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076268.069496:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076268.069519:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076268/real 1769076268] req@ffff969e0a7fe680 x1855007923716352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076268.069544:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923716352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076268.069561:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076268.069565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076268.069570:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076268.069578:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076268.069655:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076268.069667:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076268/real 1769076268] req@ffff969e0a7fed80 x1855007923716480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076268.069679:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923716480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076268.069688:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076268.069691:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076268.069694:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076268.388726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076269.092796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076269.412716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076270.116892:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076270.436656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076271.140801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076271.460707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076272.164835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076272.484703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076273.188719:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076273.188803:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076273.188810:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076273.188825:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2962 00000100:00080000:3.0:1769076273.188841:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076273.188850:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076273.188876:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076273.188918:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923716608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076273.188957:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076273.188962:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076273.188965:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076273.188970:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2962 00000100:00080000:3.0:1769076273.188975:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076273.188980:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076273.188992:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076273.189025:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923716736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076273.189038:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076273.189041:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076273.189043:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076273.189050:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2962 00000100:00100000:1.0:1769076273.189052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923716608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076273.189054:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076273.189064:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076273.189119:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076273.189142:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923716864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076273.189157:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076273.189163:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076273.189165:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076273.189170:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2962 00000100:00080000:3.0:1769076273.189177:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076273.189188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076273.189191:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076273.189202:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076273.189219:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923716992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076273.189231:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076273.189236:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076273.189239:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076273.189245:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2962 00000100:00080000:3.0:1769076273.189250:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076273.189259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076273/real 1769076273] req@ffff969e0a7fce00 x1855007923716608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076273.189278:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076273.189283:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076273.189297:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923717120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076273.189344:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923716608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076273.189401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076273.189411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076273.189419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923716608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076273.189461:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076273.189509:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076273.189549:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076273/real 1769076273] req@ffff969e0a7fdf80 x1855007923716736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076273.189568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923716736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076273.189581:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076273.189587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076273.189593:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076273.189602:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076273.189662:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076273.189702:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076273/real 1769076273] req@ffff969e0a7fca80 x1855007923716864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076273.189726:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923716864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076273.189814:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076273.189820:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076273.189826:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076273.189836:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076273.189895:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076273.189918:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076273/real 1769076273] req@ffff969e0a7fd180 x1855007923716992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076273.189942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923716992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076273.189959:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076273.189965:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076273.189971:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923716992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076273.189982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076273.190032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076273.190048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076273/real 1769076273] req@ffff969e0a7fe300 x1855007923717120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076273.190065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923717120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076273.190077:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076273.190099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076273.190120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076273.508805:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076274.212814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076274.532807:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076275.236798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076275.556785:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076276.260916:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076276.580799:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076277.284894:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076277.604753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076278.308743:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076278.308754:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076278.308759:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076278.308768:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2967 00000100:00080000:0.0:1769076278.308780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769076278.308783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076278.308788:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076278.308843:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923717248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076278.308872:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076278.308876:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076278.308878:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076278.308881:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2967 00000100:00080000:0.0:1769076278.308885:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076278.308890:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076278.308899:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076278.308911:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923717376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076278.308917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923717248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076278.308931:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076278.308935:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076278.308936:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076278.308940:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2967 00000100:00080000:0.0:1769076278.308944:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076278.308947:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076278.308951:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076278.308960:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923717504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076278.308969:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076278.308972:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076278.308974:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076278.308984:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2967 00000100:00080000:0.0:1769076278.308989:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076278.308993:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076278.308998:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076278.309000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076278.309014:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923717632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076278.309024:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076278.309028:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076278.309030:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076278.309034:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2967 00000100:00080000:0.0:1769076278.309037:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076278.309039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076278/real 1769076278] req@ffff969e06a60380 x1855007923717248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076278.309043:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076278.309052:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076278.309066:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923717760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076278.309073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923717248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076278.309087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076278.309092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076278.309097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923717248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076278.309119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076278.309138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076278.309148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076278/real 1769076278] req@ffff969e06a61180 x1855007923717376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076278.309159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923717376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076278.309166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076278.309169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076278.309171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076278.309176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076278.309190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076278.309208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076278/real 1769076278] req@ffff969e06a61500 x1855007923717504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076278.309218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923717504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076278.309226:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076278.309228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076278.309231:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076278.309236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076278.309262:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076278.309270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076278/real 1769076278] req@ffff969e06a62680 x1855007923717632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076278.309279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923717632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076278.309286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076278.309288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076278.309290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076278.309295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076278.309307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076278.309320:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076278/real 1769076278] req@ffff969e06a61f80 x1855007923717760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076278.309329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923717760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076278.309342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076278.309345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076278.309349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923717760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076278.628744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076279.332865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076279.652744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076280.356823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076280.676757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076281.380875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076281.700709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076282.404767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076282.724699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076283.428773:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076283.428788:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076283.428795:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076283.428810:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2972 00000100:00080000:2.0:1769076283.428825:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769076283.428827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076283.428838:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076283.428883:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923717888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076283.428915:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076283.428919:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076283.428922:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076283.428926:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2972 00000100:00080000:2.0:1769076283.428940:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076283.428946:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076283.428956:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076283.428970:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923717888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076283.428974:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923718016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076283.429005:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076283.429008:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076283.429010:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076283.429014:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2972 00000100:00080000:2.0:1769076283.429017:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076283.429020:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076283.429024:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076283.429070:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923718144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076283.429081:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076283.429084:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076283.429089:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076283.429093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076283.429096:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2972 00000100:00080000:2.0:1769076283.429104:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076283.429125:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076283.429131:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076283.429145:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923718272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076283.429153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076283/real 1769076283] req@ffff969e0a46ed80 x1855007923717888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076283.429160:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076283.429168:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076283.429171:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076283.429175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923717888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076283.429180:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2972 00000100:00080000:2.0:1769076283.429190:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076283.429193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076283.429197:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076283.429200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769076283.429205:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076283.429206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923717888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076283.429224:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923718400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076283.429234:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076283.429255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076283.429264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076283/real 1769076283] req@ffff969e0a46d880 x1855007923718016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076283.429272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923718016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076283.429278:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076283.429280:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076283.429283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076283.429294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076283.429309:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076283.429316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076283/real 1769076283] req@ffff969e0a46ea00 x1855007923718144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076283.429324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923718144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076283.429330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076283.429332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076283.429334:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076283.429338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076283.429349:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076283.429354:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076283/real 1769076283] req@ffff969e0a46ca80 x1855007923718272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076283.429362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923718272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076283.429368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076283.429369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076283.429371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076283.429410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076283.429427:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076283.429436:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076283/real 1769076283] req@ffff969e0a46df80 x1855007923718400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076283.429471:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923718400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076283.429480:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076283.429483:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076283.429487:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076283.748676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076284.452860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076284.772736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076285.476971:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076285.796753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076286.500836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076286.820741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076287.524803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076287.844739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076288.548779:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076288.548792:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076288.548799:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076288.548810:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2977 00000100:00080000:3.0:1769076288.548825:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076288.548834:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076288.548880:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076288.548885:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923718528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076288.548940:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076288.548945:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076288.548947:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076288.548953:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2977 00000100:00080000:3.0:1769076288.548959:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076288.548967:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076288.548981:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076288.549000:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923718656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076288.549024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923718528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076288.549024:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076288.549039:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076288.549043:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076288.549057:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2977 00000100:00080000:3.0:1769076288.549062:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076288.549090:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076288.549098:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076288.549127:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923718784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076288.549140:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076288.549144:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076288.549147:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076288.549151:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2977 00000100:00080000:3.0:1769076288.549160:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076288.549169:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076288.549176:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076288.549191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076288.549194:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923718912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076288.549211:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076288.549216:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076288.549219:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076288.549226:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2977 00000100:00080000:3.0:1769076288.549243:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076288.549251:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076288.549262:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076288.549272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076288/real 1769076288] req@ffff969e0a7fc380 x1855007923718528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076288.549281:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923719040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076288.549389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923718528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076288.549409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076288.549417:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076288.549424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923718528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076288.549467:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076288.549546:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076288.549574:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076288/real 1769076288] req@ffff969e0a7fc000 x1855007923718656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076288.549644:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923718656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076288.549687:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076288.549692:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076288.549697:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076288.549707:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076288.549741:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076288.549756:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076288/real 1769076288] req@ffff969e0a7fd500 x1855007923718784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076288.549771:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923718784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076288.549787:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076288.549794:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076288.549800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076288.549813:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076288.549854:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076288.549878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076288/real 1769076288] req@ffff969e0a7fdc00 x1855007923718912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076288.549904:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923718912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076288.549918:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076288.549923:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076288.549938:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923718912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076288.549947:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076288.549980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076288.550002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076288/real 1769076288] req@ffff969e0a7ff800 x1855007923719040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076288.550026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923719040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076288.550043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076288.550050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076288.550057:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076288.868697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076289.572851:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076289.892718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076290.596840:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076290.916730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076291.620866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076291.940710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076292.644754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076292.964703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076293.668753:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769076293.668782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076293.668794:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076293.668801:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076293.668813:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2982 00000100:00080000:3.0:1769076293.668827:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076293.668836:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076293.668880:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923719168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076293.668928:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076293.668936:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076293.668940:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076293.668950:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2982 00000100:00080000:3.0:1769076293.668958:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076293.668967:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076293.668984:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076293.668994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923719168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076293.669021:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923719296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076293.669063:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076293.669068:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076293.669071:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076293.669076:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2982 00000100:00080000:3.0:1769076293.669083:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076293.669090:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076293.669099:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076293.669115:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923719424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076293.669139:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076293.669156:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076293.669158:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076293.669179:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2982 00000800:00000100:1.0:1769076293.669188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076293.669189:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076293.669208:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076293.669228:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076293.669245:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923719552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076293.669258:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076293.669264:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076293.669268:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076293.669276:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2982 00000100:00080000:3.0:1769076293.669290:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076293.669296:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076293.669304:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076293.669310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076293/real 1769076293] req@ffff969e0a7ffb80 x1855007923719168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076293.669323:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923719680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076293.669388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923719168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076293.669465:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076293.669472:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076293.669477:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923719168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076293.669514:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076293.669548:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076293.669582:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076293/real 1769076293] req@ffff969e0a7ff800 x1855007923719296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076293.669595:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923719296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076293.669633:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076293.669638:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076293.669642:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076293.669650:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076293.669674:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076293.669685:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076293/real 1769076293] req@ffff969e0a7fdc00 x1855007923719424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076293.669697:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923719424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076293.669707:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076293.669710:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076293.669713:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076293.669720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076293.669737:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076293.669746:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076293/real 1769076293] req@ffff969e0a7fd500 x1855007923719552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076293.669756:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923719552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076293.669773:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076293.669777:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076293.669781:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076293.669787:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076293.669808:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076293.669817:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076293/real 1769076293] req@ffff969e0a7fc000 x1855007923719680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076293.669829:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923719680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076293.669854:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076293.669856:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076293.669859:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076293.988745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076294.692878:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076295.012706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076295.716829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076296.036697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076296.740818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076297.060728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076297.764822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076298.084723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076298.788730:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076298.788744:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076298.788750:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076298.788763:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2987 00000100:00080000:3.0:1769076298.788801:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076298.788810:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076298.788854:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923719808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076298.788857:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076298.788915:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076298.788920:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076298.788923:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076298.788929:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2987 00000100:00080000:3.0:1769076298.788933:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076298.788939:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076298.788952:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076298.788969:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923719936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076298.788978:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923719808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076298.788997:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076298.789002:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076298.789004:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076298.789010:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2987 00000100:00080000:3.0:1769076298.789014:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076298.789018:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076298.789025:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076298.789041:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923720064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076298.789054:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076298.789068:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076298.789072:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076298.789078:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2987 00000100:00080000:3.0:1769076298.789084:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076298.789090:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076298.789098:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076298.789128:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923720192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076298.789142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719936:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076298.789148:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076298.789154:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076298.789157:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076298.789164:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2987 00000100:00080000:3.0:1769076298.789171:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076298.789185:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076298.789193:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076298.789201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076298.789218:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923720320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076298.789263:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076298.789295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076299.108711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076299.812895:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076300.132769:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076300.836805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076301.156713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076301.860909:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076301.861020:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:1.0:1769076301.861076:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076301.861105:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076301.861171:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076301.861190:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076301.861203:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076301.861234:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:3.0:1769076301.861316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076298/real 1769076301] req@ffff969e0a7fca80 x1855007923720192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076301.861377:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923720192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076301.861401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076301.861411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076301.861446:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076301.861462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076298/real 1769076301] req@ffff969e0a7fdf80 x1855007923720320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076301.861492:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923720320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076301.861511:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076301.861518:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076301.861525:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076301.861549:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076298/real 1769076301] req@ffff969e0a7fe300 x1855007923719936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076301.861577:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923719936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076301.861653:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076301.861664:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076301.861671:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923719936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076301.861685:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076298/real 1769076301] req@ffff969e0a7fd180 x1855007923720064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076301.861710:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923720064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076301.861725:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076301.861732:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076301.861756:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076301.861776:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076298/real 1769076301] req@ffff969e0a7fc380 x1855007923719808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076301.861807:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923719808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076301.861825:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076301.861832:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076301.861839:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923719808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076302.180705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076302.884852:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076303.204700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076303.908732:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076303.908746:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076303.908753:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076303.908766:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2993 00010000:00010000:2.0:1769076303.908771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076303.908781:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076303.908794:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076303.908840:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923720448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076303.908881:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076303.908885:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076303.908888:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076303.908894:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2993 00000100:00080000:3.0:1769076303.908899:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076303.908905:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076303.908920:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076303.908955:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923720448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076303.908976:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923720576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076303.909006:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076303.909013:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076303.909016:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076303.909024:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2993 00000100:00080000:3.0:1769076303.909035:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076303.909044:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076303.909056:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076303.909068:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076303.909091:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923720704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076303.909113:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076303.909121:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076303.909126:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076303.909136:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2993 00000100:00080000:3.0:1769076303.909145:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076303.909152:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076303.909164:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076303.909166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076303/real 1769076303] req@ffff969e0a7fc380 x1855007923720448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076303.909196:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923720832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076303.909213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923720448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076303.909269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076303.909282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076303.909285:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076303.909296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923720448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076303.909298:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076303.909307:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076303.909332:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2993 00000100:00080000:3.0:1769076303.909343:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076303.909355:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720576:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076303.909357:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076303.909376:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076303.909408:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923720960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076303.909433:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076303.909469:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076303/real 1769076303] req@ffff969e0a7fd180 x1855007923720576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076303.909500:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923720576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076303.909545:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076303.909553:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076303.909561:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076303.909575:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076303.909668:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076303.909694:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076303/real 1769076303] req@ffff969e0a7fe300 x1855007923720704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076303.909719:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923720704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076303.909748:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076303.909754:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076303.909761:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076303.909774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076303.909818:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076303.909841:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076303/real 1769076303] req@ffff969e0a7fdf80 x1855007923720832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076303.909863:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923720832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076303.909880:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076303.909887:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076303.909910:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076303.909946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076303.909992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076303.910016:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076303/real 1769076303] req@ffff969e0a7fca80 x1855007923720960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076303.910039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923720960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076303.910056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076303.910062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076303.910069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923720960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076304.228711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076304.932855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076305.252702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076305.956854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076306.276741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076306.980787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076307.300779:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076308.004854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076308.324702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076309.028742:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076309.028757:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076309.028763:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076309.028776:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 2998 00000100:00080000:3.0:1769076309.028791:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076309.028801:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076309.028848:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923721088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076309.028862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076309.028888:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076309.028896:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076309.028899:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076309.028905:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2998 00000100:00080000:3.0:1769076309.028910:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076309.028915:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076309.028928:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076309.028952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923721088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076309.028952:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923721216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076309.028979:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076309.028998:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076309.029001:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076309.029007:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2998 00000100:00080000:3.0:1769076309.029014:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076309.029019:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076309.029025:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076309.029043:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923721344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076309.029057:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076309.029062:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076309.029064:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076309.029068:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2998 00000100:00080000:3.0:1769076309.029074:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076309.029084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076309.029085:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076309.029099:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076309.029115:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923721472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076309.029127:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076309.029132:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076309.029159:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076309.029167:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 2998 00000100:00100000:1.0:1769076309.029176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076309/real 1769076309] req@ffff969e0a7fce00 x1855007923721088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076309.029178:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076309.029192:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076309.029217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923721088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076309.029218:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076309.029243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076309.029250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076309.029259:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923721600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076309.029278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923721088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076309.029328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076309.029366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076309.029384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076309/real 1769076309] req@ffff969e0a7fed80 x1855007923721216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076309.029400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923721216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076309.029411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076309.029415:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076309.029419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076309.029427:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076309.029447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076309.029458:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076309/real 1769076309] req@ffff969e0a7fe680 x1855007923721344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076309.029472:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923721344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076309.029482:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076309.029485:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076309.029489:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076309.029496:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076309.029515:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076309.029539:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076309/real 1769076309] req@ffff969e0a7fc700 x1855007923721472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076309.029554:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923721472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076309.029565:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076309.029569:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076309.029573:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076309.029579:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076309.029637:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076309.029651:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076309/real 1769076309] req@ffff969e0a7fd880 x1855007923721600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076309.029666:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923721600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076309.029677:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076309.029681:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076309.029686:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076309.348718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076310.052788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076310.372665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076311.076774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076311.396715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076312.100839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076312.420702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076313.124938:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076313.444719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076314.148748:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076314.148764:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076314.148771:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076314.148786:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3003 00000100:00080000:3.0:1769076314.148802:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076314.148814:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076314.148822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076314.148862:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923721728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076314.148914:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076314.148918:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076314.148920:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076314.148924:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3003 00000100:00080000:3.0:1769076314.148929:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076314.148933:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076314.148944:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076314.148984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923721728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076314.148986:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923721856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076314.149013:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076314.149017:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076314.149020:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076314.149027:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3003 00000100:00080000:3.0:1769076314.149076:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076314.149084:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076314.149109:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076314.149126:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076314.149127:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923721984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076314.149138:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076314.149146:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076314.149148:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076314.149153:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3003 00000100:00080000:3.0:1769076314.149158:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076314.149162:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076314.149167:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076314.149180:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923722112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076314.149207:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076314.149213:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076314.149217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076314/real 1769076314] req@ffff969e0a7fea00 x1855007923721728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076314.149218:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076314.149232:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3003 00000100:00080000:3.0:1769076314.149242:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076314.149252:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076314.149259:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076314.149268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923721728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076314.149276:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923722240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076314.149301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076314.149312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076314.149327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923721728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076314.149386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076314.149415:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076314.149428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076314/real 1769076314] req@ffff969e0a7fd880 x1855007923721856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076314.149460:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923721856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076314.149470:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076314.149474:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076314.149478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076314.149485:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076314.149505:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076314.149516:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076314/real 1769076314] req@ffff969e0a7fc700 x1855007923721984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076314.149529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923721984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076314.149556:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076314.149560:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076314.149565:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923721984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076314.149572:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076314.149596:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076314.149607:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076314/real 1769076314] req@ffff969e0a7fe680 x1855007923722112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076314.149620:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923722112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076314.149630:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076314.149633:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076314.149636:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076314.149643:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076314.149708:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076314.149719:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076314/real 1769076314] req@ffff969e0a7fed80 x1855007923722240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076314.149750:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923722240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076314.149761:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076314.149764:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076314.149777:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076314.468718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076315.172849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076315.492678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076316.196914:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076316.516705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076317.220836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076317.540730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076318.244774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076318.564693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076319.268797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076319.268800:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076319.268812:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076319.268818:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076319.268830:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3008 00000100:00080000:0.0:1769076319.268843:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076319.268851:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076319.268896:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923722368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076319.268930:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076319.268953:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076319.268957:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076319.268991:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3008 00000100:00080000:0.0:1769076319.268998:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076319.269022:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076319.269033:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076319.269048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923722368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076319.269053:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923722496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076319.269073:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076319.269077:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076319.269080:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076319.269085:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3008 00000100:00080000:0.0:1769076319.269089:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076319.269093:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076319.269099:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076319.269113:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923722624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076319.269123:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076319.269127:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076319.269129:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076319.269132:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3008 00000100:00080000:0.0:1769076319.269137:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076319.269144:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076319.269146:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076319.269163:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076319.269178:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923722752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076319.269190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076319/real 1769076319] req@ffff969e06a61880 x1855007923722368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076319.269192:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076319.269219:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076319.269223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923722368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076319.269225:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076319.269240:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3008 00000100:00080000:1.0:1769076319.269246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076319.269251:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076319.269255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076319.269266:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923722368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076319.269268:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076319.269282:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076319.269301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076319.269307:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923722880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076319.269334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076319.269352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076319/real 1769076319] req@ffff969e06a63100 x1855007923722496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076319.269366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923722496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076319.269375:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076319.269379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076319.269393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076319.269400:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076319.269421:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076319.269431:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076319/real 1769076319] req@ffff969e06a61f80 x1855007923722624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076319.269443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923722624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076319.269451:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076319.269454:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076319.269457:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076319.269465:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076319.269492:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076319.269508:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076319/real 1769076319] req@ffff969e06a62680 x1855007923722752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076319.269528:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923722752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076319.269542:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076319.269548:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076319.269554:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076319.269577:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076319.269600:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076319.269649:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076319/real 1769076319] req@ffff969e06a61500 x1855007923722880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076319.269662:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923722880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076319.269672:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076319.269675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076319.269678:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923722880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076319.588690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076320.292937:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076320.612706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076321.316844:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076321.636714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076322.340820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076322.660663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076323.364845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076323.684641:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076324.388773:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076324.388788:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076324.388795:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076324.388811:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3013 00000100:00080000:3.0:1769076324.388827:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769076324.388830:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076324.388839:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076324.388937:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923723008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076324.388974:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076324.388980:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076324.388982:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076324.388988:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3013 00000100:00080000:3.0:1769076324.389002:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076324.389010:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076324.389025:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076324.389042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923723008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076324.389047:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923723136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076324.389069:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076324.389074:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076324.389077:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076324.389083:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3013 00000100:00080000:3.0:1769076324.389088:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076324.389093:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076324.389122:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076324.389139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076324.389146:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923723264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076324.389163:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076324.389168:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076324.389171:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076324.389177:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3013 00000100:00080000:3.0:1769076324.389185:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076324.389195:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076324.389196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076324/real 1769076324] req@ffff969e0a7fce00 x1855007923723008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076324.389202:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076324.389227:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923723392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076324.389237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923723008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076324.389247:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076324.389260:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076324.389267:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076324.389275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076324.389282:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3013 00000100:00080000:1.0:1769076324.389289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076324.389296:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076324.389298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923723008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076324.389315:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076324.389328:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076324.389339:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076324.389358:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923723520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076324.389377:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076324.389421:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076324/real 1769076324] req@ffff969e0a7fca80 x1855007923723136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076324.389451:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923723136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076324.389468:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076324.389474:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076324.389481:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076324.389494:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076324.389537:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076324.389558:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076324/real 1769076324] req@ffff969e0a7fdf80 x1855007923723264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076324.389575:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923723264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076324.389586:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076324.389624:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076324.389629:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076324.389638:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076324.389667:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076324.389682:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076324/real 1769076324] req@ffff969e0a7fe300 x1855007923723392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076324.389696:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923723392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076324.389717:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076324.389722:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076324.389726:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076324.389749:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076324.389774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076324.389787:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076324/real 1769076324] req@ffff969e0a7fd180 x1855007923723520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076324.389802:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923723520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076324.389812:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076324.389816:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076324.389819:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076324.708753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076325.412976:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076325.732743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076326.436904:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076326.756708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076327.461097:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076327.780715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076328.485077:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076328.804743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076329.509086:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076329.509112:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076329.509123:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076329.509149:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3018 00000100:00080000:1.0:1769076329.509174:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076329.509190:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076329.509325:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923723648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076329.509397:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076329.509406:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076329.509411:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076329.509418:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3018 00000100:00080000:1.0:1769076329.509425:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076329.509433:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076329.509469:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076329.509499:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923723776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076329.509519:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076329.509525:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076329.509529:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076329.509536:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3018 00000100:00080000:1.0:1769076329.509543:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076329.509551:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076329.509561:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076329.509587:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923723904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076329.509699:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076329.509702:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923723648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769076329.509711:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076329.509724:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076329.509757:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3018 00000100:00080000:1.0:1769076329.509767:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076329.509776:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076329.509814:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076329.509850:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923724032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076329.509872:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076329.509886:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1769076329.509889:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769076329.509895:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076329.509911:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3018 00000100:00080000:1.0:1769076329.509927:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076329.509936:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076329.509950:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076329.509980:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923724160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076329.509995:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076329/real 1769076329] req@ffff969e10b62680 x1855007923723648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076329.510047:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923723648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076329.510066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076329.510073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076329.510081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923723648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076329.510096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076329.510145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076329.510179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076329/real 1769076329] req@ffff969e10b62a00 x1855007923723776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076329.510207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923723776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076329.510226:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076329.510234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076329.510245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723776:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769076329.510249:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076329.510298:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076329.510338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076329.510356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076329/real 1769076329] req@ffff969e10b60a80 x1855007923723904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076329.510373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923723904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076329.510404:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076329.510409:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076329.510414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923723904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076329.510443:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076329.510475:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076329.510489:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076329/real 1769076329] req@ffff969e10b61f80 x1855007923724032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076329.510504:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923724032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076329.510515:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076329.510519:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076329.510523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076329.510531:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076329.510550:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076329.510562:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076329/real 1769076329] req@ffff969e10b61500 x1855007923724160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076329.510585:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923724160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076329.510656:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076329.510663:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076329.510683:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076329.828773:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076330.533542:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076330.852827:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076331.556983:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076331.876671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076332.580894:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076332.900692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076333.604855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076333.924707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076334.628777:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076334.628794:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076334.628800:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076334.628815:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3023 00000100:00080000:1.0:1769076334.628830:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076334.628861:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076334.628910:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923724288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076334.628961:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076334.628969:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076334.628972:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076334.628980:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3023 00000100:00080000:1.0:1769076334.628988:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076334.628998:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076334.629014:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076334.629039:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923724416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076334.629071:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076334.629075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923724288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769076334.629079:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076334.629090:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076334.629102:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3023 00000100:00080000:1.0:1769076334.629108:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076334.629116:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076334.629128:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076334.629152:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923724544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076334.629170:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076334.629177:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076334.629191:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076334.629199:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3023 00000800:00000100:3.0:1769076334.629208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769076334.629210:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076334.629229:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076334.629241:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076334.629269:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923724672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076334.629286:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076334/real 1769076334] req@ffff969e10b61500 x1855007923724288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076334.629293:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076334.629298:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076334.629304:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076334.629320:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3023 00000100:00080000:1.0:1769076334.629333:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076334.629345:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076334.629357:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076334.629380:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923724288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076334.629382:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923724800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076334.629480:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076334.629497:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076334.629510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923724288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076334.629571:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724416:192.168.204.143@tcp:38:kworker.0 00010000:00010000:0.0:1769076334.629663:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1769076334.629689:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076334.629739:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076334/real 1769076334] req@ffff969e10b61f80 x1855007923724416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076334.629759:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923724416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076334.629771:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076334.629775:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076334.629780:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076334.629790:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076334.629819:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076334.629857:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076334/real 1769076334] req@ffff969e10b60a80 x1855007923724544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076334.629873:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923724544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076334.629884:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076334.629889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076334.629893:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076334.629901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076334.629926:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076334.629937:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076334/real 1769076334] req@ffff969e10b62a00 x1855007923724672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076334.629952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923724672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076334.629962:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076334.629966:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076334.629969:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076334.629977:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076334.629998:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076334.630009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076334/real 1769076334] req@ffff969e10b62680 x1855007923724800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076334.630023:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923724800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076334.630043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076334.630047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076334.630052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923724800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076334.948765:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076335.652870:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076335.972709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076336.676939:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076336.996745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076337.700796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076338.020755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076338.724867:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076339.044714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076339.748822:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076339.748836:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076339.748842:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769076339.748852:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076339.748856:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3028 00000100:00080000:3.0:1769076339.748874:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076339.748884:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076339.748931:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923724928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076339.748971:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076339.748976:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076339.748978:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076339.748983:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3028 00000100:00080000:3.0:1769076339.748987:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076339.748992:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076339.749003:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076339.749020:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923725056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076339.749035:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076339.749039:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076339.749055:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923724928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076339.749066:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076339.749076:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3028 00000100:00080000:3.0:1769076339.749081:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076339.749087:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076339.749097:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076339.749116:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923725184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076339.749129:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076339.749134:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076339.749137:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076339.749142:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3028 00000100:00080000:3.0:1769076339.749146:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076339.749150:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076339.749159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769076339.749161:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076339.749180:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923725312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076339.749193:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076339.749198:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076339.749200:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076339.749205:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3028 00000100:00080000:3.0:1769076339.749210:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076339.749216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076339/real 1769076339] req@ffff969e0a7fc380 x1855007923724928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076339.749220:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076339.749233:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076339.749264:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923725440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076339.749283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923724928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076339.749301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076339.749308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076339.749315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923724928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076339.749347:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076339.749378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076339.749394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076339/real 1769076339] req@ffff969e0a7fc000 x1855007923725056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076339.749409:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923725056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076339.749420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076339.749424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076339.749429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076339.749436:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076339.749454:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076339.749465:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076339/real 1769076339] req@ffff969e0a7fd500 x1855007923725184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076339.749479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923725184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076339.749507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076339.749511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076339.749516:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076339.749523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076339.749549:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076339.749561:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076339/real 1769076339] req@ffff969e0a7fdc00 x1855007923725312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076339.749575:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923725312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076339.749585:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076339.749652:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076339.749658:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076339.749666:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076339.749697:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076339.749712:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076339/real 1769076339] req@ffff969e0a7ff800 x1855007923725440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076339.749727:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923725440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076339.749738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076339.749742:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076339.749757:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076340.068745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076340.772930:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076341.092714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076341.796790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076342.116680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076342.820770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076343.140693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076343.844816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076344.164712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076344.868715:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076344.868729:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076344.868735:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076344.868749:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3034 00000100:00080000:3.0:1769076344.868764:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076344.868775:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076344.868806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076344.868828:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923725568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076344.868864:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076344.868870:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076344.868873:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076344.868881:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3034 00000100:00080000:3.0:1769076344.868888:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076344.868894:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076344.868908:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076344.868933:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923725696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076344.868936:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923725568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076344.868951:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076344.868956:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076344.868959:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076344.868965:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3034 00000100:00080000:3.0:1769076344.868970:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076344.868996:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076344.869004:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076344.869021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076344.869021:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923725824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076344.869036:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076344.869041:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076344.869044:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076344.869050:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3034 00000100:00080000:3.0:1769076344.869055:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076344.869062:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076344.869067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076344/real 1769076344] req@ffff969e0a7ffb80 x1855007923725568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076344.869070:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076344.869085:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923725952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076344.869095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923725568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076344.869100:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076344.869106:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076344.869109:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076344.869112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076344.869114:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3034 00000100:00080000:1.0:1769076344.869119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076344.869120:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076344.869126:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076344.869127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923725568:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076344.869133:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076344.869146:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923726080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076344.869160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076344.869196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076344.869219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076344/real 1769076344] req@ffff969e0a7ff800 x1855007923725696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076344.869237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923725696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076344.869247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076344.869251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076344.869264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076344.869271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076344.869317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076344.869331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076344/real 1769076344] req@ffff969e0a7fdc00 x1855007923725824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076344.869346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923725824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076344.869356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076344.869360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076344.869364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076344.869370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076344.869405:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076344.869416:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076344/real 1769076344] req@ffff969e0a7fd500 x1855007923725952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076344.869428:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923725952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076344.869437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076344.869439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076344.869443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923725952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076344.869459:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076344.869476:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076344.869511:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076344/real 1769076344] req@ffff969e0a7fc000 x1855007923726080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076344.869525:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923726080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076344.869555:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076344.869558:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076344.869562:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076345.188688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076345.892860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076346.212707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076346.916834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076347.236713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076347.940837:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076348.260683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076348.964833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076349.284719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076349.988727:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076349.988783:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076349.988792:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076349.988813:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3039 00010000:00010000:2.0:1769076349.988833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076349.988837:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076349.988859:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076349.988933:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923726208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076349.988976:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076349.988981:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076349.988984:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076349.988991:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3039 00000100:00080000:3.0:1769076349.988997:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076349.989004:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076349.989020:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076349.989039:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923726336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076349.989049:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923726208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076349.989061:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076349.989067:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076349.989069:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076349.989075:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3039 00000100:00080000:3.0:1769076349.989079:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076349.989083:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076349.989088:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076349.989105:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923726464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076349.989132:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076349.989139:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076349.989143:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076349.989151:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3039 00000100:00080000:3.0:1769076349.989161:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076349.989170:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076349.989181:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076349.989190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076349.989214:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923726592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076349.989238:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076349.989246:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076349.989249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076349/real 1769076349] req@ffff969e0a7fc380 x1855007923726208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076349.989253:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076349.989270:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3039 00000100:00080000:3.0:1769076349.989281:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076349.989290:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076349.989299:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076349.989308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923726208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076349.989323:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923726720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076349.989348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076349.989364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076349.989378:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923726208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076349.989426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076349.989514:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076349.989549:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076349/real 1769076349] req@ffff969e0a7fd180 x1855007923726336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076349.989568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923726336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076349.989580:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076349.989585:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076349.989625:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076349.989634:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076349.989663:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076349.989678:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076349/real 1769076349] req@ffff969e0a7fe300 x1855007923726464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076349.989694:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923726464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076349.989704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076349.989709:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076349.989713:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076349.989720:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076349.989741:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076349.989759:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076349/real 1769076349] req@ffff969e0a7fdf80 x1855007923726592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076349.989782:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923726592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076349.989817:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076349.989826:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076349.989834:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076349.989848:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076349.989904:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076349.989921:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076349/real 1769076349] req@ffff969e0a7fca80 x1855007923726720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076349.989936:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923726720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076349.989948:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076349.989952:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076349.989956:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076350.308744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076351.013046:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076351.332646:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076352.036751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076352.356712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076353.060773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076353.380668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076354.084828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076354.404732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076355.108755:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076355.108769:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076355.108776:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076355.108788:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3044 00000100:00080000:3.0:1769076355.108828:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076355.108839:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076355.108844:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076355.108886:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923726848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076355.108928:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076355.108932:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076355.108935:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076355.108941:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3044 00000100:00080000:3.0:1769076355.108945:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076355.108950:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076355.108962:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076355.108978:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923726976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076355.108993:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076355.108997:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076355.109000:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076355.109005:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3044 00000100:00080000:3.0:1769076355.109009:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076355.109019:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076355.109020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923726848:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076355.109026:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076355.109043:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923727104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076355.109057:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076355.109075:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076355.109079:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076355.109085:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3044 00000100:00080000:3.0:1769076355.109091:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076355.109097:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076355.109105:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076355.109118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076355.109128:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923727232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076355.109145:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076355.109150:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076355.109152:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076355.109160:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3044 00000100:00080000:3.0:1769076355.109164:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076355.109168:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076355.109175:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076355.109187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076355/real 1769076355] req@ffff969e0a7fce00 x1855007923726848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076355.109208:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923727360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076355.109222:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923726848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076355.109256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076355.109263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076355.109270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923726848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076355.109281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076355.109315:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076355.109332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076355/real 1769076355] req@ffff969e0a7fed80 x1855007923726976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076355.109348:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923726976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076355.109386:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076355.109392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076355.109397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923726976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076355.109431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076355.109460:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076355.109475:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076355/real 1769076355] req@ffff969e0a7fe680 x1855007923727104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076355.109490:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923727104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076355.109500:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076355.109505:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076355.109509:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076355.109527:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076355.109553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076355.109566:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076355/real 1769076355] req@ffff969e0a7fc700 x1855007923727232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076355.109580:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923727232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076355.109623:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076355.109627:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076355.109631:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076355.109639:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076355.109662:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076355.109675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076355/real 1769076355] req@ffff969e0a7fd880 x1855007923727360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076355.109690:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923727360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076355.109700:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076355.109704:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076355.109708:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076355.428734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076356.132835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076356.452723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076357.156703:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076357.476646:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076358.180743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076358.500754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076359.204846:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076359.524732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076360.228827:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076360.228844:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076360.228868:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076360.228886:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3049 00000100:00080000:2.0:1769076360.228905:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769076360.228913:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076360.228922:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076360.229002:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923727488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076360.229039:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076360.229044:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076360.229047:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076360.229052:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3049 00000100:00080000:2.0:1769076360.229058:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076360.229064:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076360.229081:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076360.229095:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923727488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076360.229101:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923727616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076360.229173:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076360.229179:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076360.229183:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076360.229195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076360.229204:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3049 00000100:00080000:2.0:1769076360.229219:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076360.229229:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076360.229240:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076360.229243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076360/real 1769076360] req@ffff969e0a46ce00 x1855007923727488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076360.229265:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923727744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076360.229272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923727488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076360.229329:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076360.229343:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076360.229349:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076360.229350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076360.229358:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3049 00000100:00080000:2.0:1769076360.229370:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076360.229371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076360.229385:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076360.229396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923727488:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076360.229399:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076360.229428:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923727872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076360.229441:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076360.229445:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076360.229450:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076360.229460:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3049 00000100:00100000:1.0:1769076360.229463:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727616:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076360.229484:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076360.229495:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076360.229502:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076360.229522:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923728000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076360.229536:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076360.229578:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076360/real 1769076360] req@ffff969e0a46e300 x1855007923727616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076360.229643:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923727616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076360.229669:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076360.229676:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076360.229681:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076360.229692:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076360.229718:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076360.229731:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076360/real 1769076360] req@ffff969e0a46df80 x1855007923727744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076360.229760:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923727744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076360.229769:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076360.229772:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076360.229775:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076360.229782:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076360.229802:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076360.229812:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076360/real 1769076360] req@ffff969e0a46ca80 x1855007923727872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076360.229823:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923727872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076360.229831:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076360.229833:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076360.229836:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923727872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076360.229854:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076360.229869:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076360.229878:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076360/real 1769076360] req@ffff969e0a46ea00 x1855007923728000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076360.229888:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923728000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076360.229910:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076360.229913:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076360.229917:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076360.548803:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076361.253115:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076361.572693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076362.276831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076362.596792:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076363.300901:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076363.620697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076364.324961:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076364.644693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076365.348846:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076365.348872:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076365.348879:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076365.348892:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3054 00000100:00080000:3.0:1769076365.348906:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076365.348916:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076365.348929:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076365.348966:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923728128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076365.349028:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076365.349033:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076365.349036:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076365.349042:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3054 00000100:00080000:3.0:1769076365.349047:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076365.349066:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076365.349083:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076365.349114:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923728256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076365.349150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923728128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076365.349157:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076365.349178:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076365.349183:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076365.349247:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3054 00000100:00080000:3.0:1769076365.349254:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076365.349262:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076365.349276:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076365.349306:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923728384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076365.349329:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076365.349336:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076365.349340:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076365.349352:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3054 00000100:00080000:3.0:1769076365.349379:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076365.349399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728256:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076365.349399:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076365.349426:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076365.349455:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923728512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076365.349477:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076365.349482:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076365.349485:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076365.349492:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3054 00000100:00080000:3.0:1769076365.349502:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076365.349508:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728384:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076365.349521:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076365.349536:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076365.349559:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076365.349573:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923728640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076365.349708:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076365.668687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076366.372826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076366.692715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076367.396867:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076367.716663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076368.420801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769076368.420939:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769076368.420985:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076368.421009:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076368.421057:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076368.421070:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076368.421077:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076368.421085:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076368.421245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076365/real 1769076368] req@ffff969e0a7fed80 x1855007923728640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076368.421308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923728640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076368.421333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076368.421344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076368.421407:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076368.421425:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076365/real 1769076368] req@ffff969e0a7fe680 x1855007923728512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076368.421454:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923728512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076368.421471:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076368.421478:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076368.421486:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076368.421502:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076365/real 1769076368] req@ffff969e0a7fd880 x1855007923728256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076368.421528:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923728256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076368.421545:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076368.421552:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076368.421559:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076368.421575:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076365/real 1769076368] req@ffff969e0a7fc700 x1855007923728384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076368.421647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923728384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076368.421665:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076368.421675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076368.421697:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076368.421716:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076365/real 1769076368] req@ffff969e0a7fea00 x1855007923728128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076368.421746:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923728128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076368.421761:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076368.421769:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076368.421775:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923728128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076368.740727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076369.444873:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076369.764716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076370.468746:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076370.468759:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076370.468766:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076370.468778:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3059 00000100:00080000:3.0:1769076370.468793:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076370.468804:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076370.468850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076370.468863:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923728768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076370.468909:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076370.468938:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076370.468941:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076370.468949:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3059 00000100:00080000:3.0:1769076370.468957:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076370.468966:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076370.468981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076370.469018:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923728896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076370.469025:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923728768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076370.469066:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076370.469070:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076370.469072:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076370.469081:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3059 00000100:00080000:3.0:1769076370.469087:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076370.469093:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076370.469099:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076370.469152:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923729024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076370.469163:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076370.469170:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076370.469172:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076370.469176:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3059 00000100:00080000:3.0:1769076370.469181:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076370.469184:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076370.469204:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076370.469230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076370.469234:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923729152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076370.469257:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076370.469273:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076370.469277:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076370.469285:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3059 00000100:00080000:3.0:1769076370.469295:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076370.469302:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076370.469311:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076370.469331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076370/real 1769076370] req@ffff969e0a7fce00 x1855007923728768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076370.469336:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923729280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076370.469438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923728768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076370.469468:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076370.469479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076370.469487:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923728768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076370.469549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076370.469650:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076370.469680:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076370/real 1769076370] req@ffff969e0a7fca80 x1855007923728896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076370.469737:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923728896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076370.469757:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076370.469765:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076370.469771:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923728896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076370.469785:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076370.469834:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076370.469859:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076370/real 1769076370] req@ffff969e0a7fdf80 x1855007923729024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076370.469885:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923729024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076370.469903:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076370.469910:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076370.469916:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076370.469929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076370.469970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076370.469994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076370/real 1769076370] req@ffff969e0a7fe300 x1855007923729152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076370.470017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923729152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076370.470060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076370.470070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076370.470077:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076370.470090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076370.470176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076370.470204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076370/real 1769076370] req@ffff969e0a7fd180 x1855007923729280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076370.470229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923729280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076370.470248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076370.470256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076370.470263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076370.788784:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076371.493059:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076371.812789:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076372.516917:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076372.836781:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076373.540869:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076373.860704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076374.564949:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076374.884758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076375.588726:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076375.588737:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076375.588742:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076375.588772:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3064 00000100:00080000:2.0:1769076375.588786:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076375.588798:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076375.588846:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923729408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076375.588886:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076375.588900:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076375.588906:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076375.588909:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076375.588914:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3064 00000100:00080000:2.0:1769076375.588935:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076375.588944:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076375.588957:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076375.588987:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923729536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076375.588998:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923729408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076375.589005:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076375.589038:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076375.589042:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076375.589053:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3064 00000100:00080000:2.0:1769076375.589060:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076375.589084:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076375.589092:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076375.589111:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923729664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076375.589123:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076375.589127:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076375.589129:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076375.589134:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3064 00000100:00080000:2.0:1769076375.589140:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076375.589146:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076375.589152:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076375.589168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076375.589176:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923729792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076375.589208:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076375.589213:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076375.589216:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076375.589223:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3064 00000100:00080000:2.0:1769076375.589230:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076375.589231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076375/real 1769076375] req@ffff969e0a46d880 x1855007923729408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076375.589240:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076375.589254:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076375.589297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923729408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076375.589302:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923729920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076375.589341:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076375.589360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076375.589368:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923729408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076375.589415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076375.589455:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076375.589482:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076375/real 1769076375] req@ffff969e0a46ed80 x1855007923729536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076375.589509:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923729536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076375.589528:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076375.589536:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076375.589543:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076375.589557:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076375.589630:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076375.589651:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076375/real 1769076375] req@ffff969e0a46f480 x1855007923729664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076375.589668:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923729664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076375.589679:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076375.589683:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076375.589688:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076375.589711:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076375.589742:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076375.589755:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076375/real 1769076375] req@ffff969e0a46d500 x1855007923729792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076375.589770:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923729792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076375.589781:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076375.589786:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076375.589790:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076375.589797:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076375.589816:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076375.589826:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076375/real 1769076375] req@ffff969e0a46c380 x1855007923729920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076375.589840:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923729920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076375.589854:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076375.589860:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076375.589867:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923729920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076375.908700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076376.612882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076376.932724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076377.636899:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076377.956733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076378.660854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076378.980742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076379.684914:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076380.004762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076380.708776:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076380.708796:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076380.708804:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076380.708822:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3069 00010000:00010000:0.0:1769076380.708841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076380.708841:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076380.708866:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076380.708961:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923730048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076380.709010:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076380.709015:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076380.709018:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076380.709024:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3069 00000100:00080000:3.0:1769076380.709058:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076380.709066:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076380.709083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923730048:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076380.709085:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076380.709119:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923730176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076380.709139:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076380.709144:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076380.709147:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076380.709154:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3069 00000100:00080000:3.0:1769076380.709160:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076380.709165:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076380.709187:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076380.709198:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076380.709214:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923730304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076380.709233:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076380.709239:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076380.709242:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076380.709254:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3069 00000100:00080000:3.0:1769076380.709261:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076380.709271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076380/real 1769076380] req@ffff969e0a7fc380 x1855007923730048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076380.709279:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076380.709300:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076380.709317:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923730432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076380.709349:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923730048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076380.709374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076380.709388:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076380.709389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076380.709405:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076380.709408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923730048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076380.709412:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076380.709427:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3069 00000100:00080000:3.0:1769076380.709436:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076380.709441:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076380.709455:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076380.709460:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076380.709488:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923730560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076380.709508:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076380.709540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076380/real 1769076380] req@ffff969e0a7fc000 x1855007923730176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076380.709557:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923730176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076380.709570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076380.709575:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076380.709579:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076380.709621:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076380.709646:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076380.709660:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076380/real 1769076380] req@ffff969e0a7fd500 x1855007923730304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076380.709697:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923730304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076380.709708:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076380.709712:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076380.709716:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076380.709725:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076380.709750:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076380.709762:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076380/real 1769076380] req@ffff969e0a7fdc00 x1855007923730432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076380.709776:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923730432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076380.709786:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076380.709790:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076380.709793:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076380.709815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076380.709834:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076380.709845:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076380/real 1769076380] req@ffff969e0a7ff800 x1855007923730560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076380.709859:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923730560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076380.709886:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076380.709891:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076380.709895:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076381.028774:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076381.732891:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076382.052707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076382.756807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076383.076674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076383.780774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076384.100697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076384.804959:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076385.124670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076385.828720:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076385.828734:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076385.828740:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076385.828752:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3075 00000100:00080000:3.0:1769076385.828777:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076385.828788:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076385.828837:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923730688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076385.828897:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076385.828902:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076385.828906:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769076385.828909:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076385.828914:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3075 00000100:00080000:3.0:1769076385.828925:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076385.828931:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076385.828967:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076385.828992:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923730816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076385.828997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923730688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076385.829018:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076385.829025:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076385.829028:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076385.829033:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3075 00000100:00080000:3.0:1769076385.829040:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076385.829045:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076385.829052:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076385.829067:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923730944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076385.829081:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076385.829085:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076385.829087:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076385.829092:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3075 00000100:00080000:3.0:1769076385.829108:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076385.829113:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076385.829129:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076385.829140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076385.829163:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923731072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076385.829179:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076385.829183:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076385.829202:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076385.829208:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3075 00000100:00080000:3.0:1769076385.829215:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076385.829223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076385/real 1769076385] req@ffff969e0a7ffb80 x1855007923730688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076385.829226:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076385.829237:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076385.829260:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923731200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076385.829278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923730688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076385.829302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076385.829312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076385.829322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923730688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076385.829372:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076385.829416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076385.829440:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076385/real 1769076385] req@ffff969e0a7ff800 x1855007923730816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076385.829464:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923730816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076385.829516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076385.829524:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076385.829530:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076385.829543:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076385.829585:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076385.829650:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076385/real 1769076385] req@ffff969e0a7fdc00 x1855007923730944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076385.829679:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923730944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076385.829696:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076385.829703:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076385.829709:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923730944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076385.829722:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076385.829761:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076385.829792:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076385/real 1769076385] req@ffff969e0a7fd500 x1855007923731072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076385.829817:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923731072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076385.829831:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076385.829835:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076385.829885:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076385.829899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076385.829945:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076385.829969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076385/real 1769076385] req@ffff969e0a7fc000 x1855007923731200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076385.829995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923731200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076385.830012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076385.830018:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076385.830023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076386.148690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076386.852920:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076387.172721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076387.876907:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076388.196723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076388.900948:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076389.220735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076389.924873:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076390.244722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076390.948826:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769076390.948833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076390.948847:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076390.948856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076390.948874:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3080 00000100:00080000:0.0:1769076390.948909:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076390.948920:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076390.948969:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923731328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076390.949033:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076390.949038:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076390.949041:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076390.949047:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3080 00000100:00080000:0.0:1769076390.949055:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076390.949063:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076390.949077:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076390.949098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923731328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076390.949102:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923731456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076390.949125:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076390.949129:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076390.949133:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076390.949138:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3080 00000100:00080000:0.0:1769076390.949142:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076390.949148:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076390.949154:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076390.949169:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923731584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076390.949184:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076390.949188:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076390.949191:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076390.949195:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3080 00000800:00000100:1.0:1769076390.949206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076390.949222:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076390.949232:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076390.949240:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076390.949259:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923731712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076390.949273:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076390.949279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076390/real 1769076390] req@ffff969e06a61180 x1855007923731328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076390.949280:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076390.949287:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076390.949302:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3080 00000100:00080000:0.0:1769076390.949314:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076390.949333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923731328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076390.949336:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076390.949348:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076390.949363:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923731840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076390.949385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076390.949401:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076390.949427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923731328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076390.949469:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076390.949516:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076390.949536:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076390/real 1769076390] req@ffff969e06a60380 x1855007923731456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076390.949558:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923731456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076390.949575:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076390.949581:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076390.949629:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076390.949645:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076390.949683:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076390.949700:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076390/real 1769076390] req@ffff969e06a62d80 x1855007923731584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076390.949716:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923731584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076390.949727:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076390.949731:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076390.949736:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076390.949744:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076390.949799:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076390.949828:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076390/real 1769076390] req@ffff969e06a60000 x1855007923731712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076390.949844:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923731712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076390.949855:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076390.949859:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076390.949863:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076390.949871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076390.949895:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076390.949906:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076390/real 1769076390] req@ffff969e06a60700 x1855007923731840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076390.949920:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923731840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076390.949930:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076390.949933:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076390.949937:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923731840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076391.268727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076391.972882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076392.292688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076392.996810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076393.316803:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076394.021059:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076394.340743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076395.045250:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076395.364771:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076396.069010:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076396.069060:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076396.069071:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076396.069094:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3085 00000100:00080000:2.0:1769076396.069118:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076396.069135:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076396.069220:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923731968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076396.069286:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076396.069295:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076396.069324:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076396.069335:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3085 00000100:00080000:2.0:1769076396.069343:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076396.069351:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076396.069371:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076396.069401:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923732096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076396.069425:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076396.069431:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076396.069435:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076396.069444:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3085 00000100:00080000:2.0:1769076396.069452:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076396.069460:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076396.069471:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076396.069497:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923732224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076396.069518:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076396.069525:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076396.069529:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076396.069548:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3085 00000100:00080000:2.0:1769076396.069556:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076396.069564:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076396.069574:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076396.069718:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923732352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076396.069743:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076396.069752:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076396.069756:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076396.069765:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3085 00000100:00080000:2.0:1769076396.069777:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076396.069785:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076396.069794:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076396.069822:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923732480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076396.069834:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923731968:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769076396.069974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076396.070064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076396/real 1769076396] req@ffff969e0a46f100 x1855007923731968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076396.070102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923731968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076396.070148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076396.070158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076396.070166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923731968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076396.070185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076396.070244:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076396.070277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076396/real 1769076396] req@ffff969e0a46f800 x1855007923732096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076396.070296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923732096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076396.070308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076396.070312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076396.070317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076396.070326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076396.070351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076396.070363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076396/real 1769076396] req@ffff969e0a46c380 x1855007923732224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076396.070378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923732224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076396.070408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076396.070413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076396.070417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076396.070425:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076396.070452:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076396.070465:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076396/real 1769076396] req@ffff969e0a46d500 x1855007923732352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076396.070497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923732352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076396.070507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076396.070511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076396.070514:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076396.070554:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076396.070591:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076396.070702:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076396/real 1769076396] req@ffff969e0a46f480 x1855007923732480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076396.070740:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923732480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076396.070752:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076396.070756:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076396.070771:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732480:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769076396.070926:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076396.388788:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076397.092984:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076397.412809:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076398.117145:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076398.436767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076399.140854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076399.460744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076400.164830:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076400.484745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076401.188786:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076401.188804:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076401.188811:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769076401.188823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076401.188830:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3090 00000100:00080000:3.0:1769076401.188850:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076401.188864:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076401.188919:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923732608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076401.188958:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076401.188963:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076401.188966:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076401.188972:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3090 00000100:00080000:3.0:1769076401.188977:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076401.188983:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076401.188996:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076401.189014:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923732736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076401.189022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923732608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076401.189076:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076401.189083:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076401.189104:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076401.189114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076401.189124:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3090 00000100:00080000:3.0:1769076401.189137:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076401.189147:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076401.189161:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076401.189167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076401/real 1769076401] req@ffff969e0a7fc380 x1855007923732608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076401.189190:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923732864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076401.189201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923732608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076401.189224:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076401.189231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076401.189233:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076401.189242:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076401.189249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076401.189257:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3090 00000100:00100000:1.0:1769076401.189264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923732608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076401.189269:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076401.189280:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076401.189288:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076401.189304:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923732992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076401.189308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732736:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076401.189336:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076401.189342:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076401.189345:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076401.189353:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3090 00000800:00000100:1.0:1769076401.189357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076401.189374:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076401.189380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076401/real 1769076401] req@ffff969e0a7fd180 x1855007923732736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076401.189388:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076401.189402:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076401.189413:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923732736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076401.189425:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923733120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076401.189433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076401.189449:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076401.189475:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076401.189486:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076401.189516:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076401.189529:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076401/real 1769076401] req@ffff969e0a7fe300 x1855007923732864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076401.189544:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923732864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076401.189554:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076401.189558:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076401.189561:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076401.189580:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076401.189682:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076401.189705:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076401/real 1769076401] req@ffff969e0a7fdf80 x1855007923732992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076401.189721:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923732992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076401.189732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076401.189736:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076401.189740:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923732992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076401.189748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076401.189771:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076401.189797:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076401/real 1769076401] req@ffff969e0a7fca80 x1855007923733120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076401.189812:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923733120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076401.189823:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076401.189827:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076401.189830:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076401.508708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076402.212813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076402.532763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076403.236836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076403.556689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076404.260754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076404.580696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076405.284704:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076405.604694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076406.308812:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076406.308831:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076406.308839:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076406.308855:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3095 00000100:00080000:3.0:1769076406.308876:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076406.308889:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076406.308907:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076406.308968:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923733248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076406.309027:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076406.309033:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076406.309036:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076406.309043:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3095 00000100:00080000:3.0:1769076406.309051:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076406.309058:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076406.309087:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076406.309093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923733248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076406.309122:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923733376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076406.309141:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076406.309187:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076406.309191:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076406.309197:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3095 00000100:00080000:3.0:1769076406.309204:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076406.309208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076406.309217:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076406.309229:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076406.309252:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923733504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076406.309269:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076406.309270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076406/real 1769076406] req@ffff969e0a7fce00 x1855007923733248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076406.309294:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076406.309306:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076406.309347:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3095 00000100:00100000:1.0:1769076406.309357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923733248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076406.309358:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076406.309377:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076406.309399:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076406.309402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076406.309442:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076406.309444:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923733632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076406.309471:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923733248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076406.309494:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076406.309536:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076406.309540:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076406.309548:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3095 00000100:00080000:3.0:1769076406.309556:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076406.309562:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076406.309577:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733376:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769076406.309647:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076406.309672:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076406.309679:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923733760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076406.309747:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076406/real 1769076406] req@ffff969e0a7fea00 x1855007923733376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076406.309765:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923733376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076406.309775:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076406.309779:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076406.309783:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076406.309791:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076406.309814:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076406.309831:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076406/real 1769076406] req@ffff969e0a7fc700 x1855007923733504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076406.309852:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923733504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076406.309865:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076406.309872:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076406.309878:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076406.309905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076406.309939:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076406.309954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076406/real 1769076406] req@ffff969e0a7fd880 x1855007923733632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076406.309967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923733632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076406.309986:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076406.309990:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076406.309994:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076406.310015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076406.310037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076406.310049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076406/real 1769076406] req@ffff969e0a7fe680 x1855007923733760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076406.310061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923733760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076406.310070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076406.310073:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076406.310076:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923733760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076406.628683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076407.332801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076407.652702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076408.356876:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076408.676747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076409.380853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076409.700698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076410.404817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076410.724716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076411.428798:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076411.428817:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076411.428825:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076411.428840:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3100 00000100:00080000:3.0:1769076411.428856:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076411.428868:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076411.428879:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076411.428922:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923733888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076411.428963:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076411.428968:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076411.428971:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076411.428977:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3100 00000100:00080000:3.0:1769076411.428981:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076411.428987:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076411.429000:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076411.429018:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923734016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076411.429030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923733888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076411.429039:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076411.429046:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076411.429048:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076411.429054:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3100 00000100:00080000:3.0:1769076411.429074:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076411.429082:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076411.429089:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076411.429107:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923734144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076411.429121:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076411.429130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076411.429132:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076411.429139:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076411.429146:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3100 00000100:00080000:3.0:1769076411.429153:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076411.429159:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076411.429167:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076411.429185:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923734272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076411.429193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076411/real 1769076411] req@ffff969e0a7fed80 x1855007923733888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076411.429207:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076411.429223:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076411.429230:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076411.429239:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3100 00000100:00080000:3.0:1769076411.429245:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076411.429252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923733888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076411.429256:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076411.429262:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076411.429278:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923734400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076411.429294:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076411.429304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076411.429319:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923733888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076411.429365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076411.429404:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076411.429481:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076411/real 1769076411] req@ffff969e0a7fe680 x1855007923734016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076411.429509:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923734016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076411.429527:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076411.429535:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076411.429542:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076411.429555:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076411.429625:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076411.429644:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076411/real 1769076411] req@ffff969e0a7fd880 x1855007923734144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076411.429659:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923734144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076411.429675:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076411.429682:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076411.429688:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076411.429714:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076411.429757:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076411.429780:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076411/real 1769076411] req@ffff969e0a7fc700 x1855007923734272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076411.429803:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923734272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076411.429815:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076411.429819:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076411.429823:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076411.429830:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076411.429855:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076411.429868:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076411/real 1769076411] req@ffff969e0a7fea00 x1855007923734400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076411.429882:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923734400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076411.429892:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076411.429895:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076411.429899:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076411.748705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076412.452910:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076412.772724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076413.476939:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076413.796680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076414.500795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076414.820691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076415.524902:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076415.844685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076416.548833:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076416.548846:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076416.548854:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769076416.548855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076416.548872:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3105 00000100:00080000:2.0:1769076416.548887:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076416.548896:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076416.548942:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923734528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076416.548983:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076416.548988:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076416.548990:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076416.548996:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3105 00000100:00080000:2.0:1769076416.549000:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076416.549005:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076416.549018:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076416.549033:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923734656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076416.549048:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076416.549055:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076416.549061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923734528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076416.549062:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076416.549072:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3105 00000100:00080000:2.0:1769076416.549083:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076416.549111:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076416.549119:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076416.549139:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923734784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076416.549152:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076416.549157:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076416.549159:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076416.549164:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3105 00000100:00080000:2.0:1769076416.549170:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076416.549174:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076416.549184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769076416.549185:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076416.549225:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923734912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076416.549239:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076416.549241:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076416/real 1769076416] req@ffff969e0a46ed80 x1855007923734528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076416.549249:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076416.549256:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076416.549272:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3105 00000100:00080000:2.0:1769076416.549282:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076416.549288:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076416.549296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923734528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076416.549299:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076416.549322:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923735040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076416.549323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076416.549373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076416.549384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923734528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076416.549427:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076416.549478:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076416.549501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076416/real 1769076416] req@ffff969e0a46d880 x1855007923734656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076416.549518:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923734656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076416.549530:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076416.549534:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076416.549539:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076416.549547:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076416.549571:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076416.549583:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076416/real 1769076416] req@ffff969e0a46ea00 x1855007923734784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076416.549651:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923734784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076416.549663:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076416.549667:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076416.549671:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076416.549679:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076416.549708:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076416.549721:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076416/real 1769076416] req@ffff969e0a46ca80 x1855007923734912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076416.549736:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923734912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076416.549747:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076416.549751:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076416.549755:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923734912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076416.549761:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076416.549779:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076416.549789:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076416/real 1769076416] req@ffff969e0a46df80 x1855007923735040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076416.549803:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923735040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076416.549831:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076416.549835:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076416.549839:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076416.868726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076417.572887:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076417.892694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076418.596836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076418.916780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076419.620820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076419.940750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076420.644819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076420.964716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076421.668824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076421.668824:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076421.668839:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076421.668846:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076421.668859:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3110 00000100:00080000:3.0:1769076421.668872:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076421.668881:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076421.668930:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923735168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076421.668971:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076421.668976:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076421.668978:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076421.668983:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3110 00000100:00080000:3.0:1769076421.668988:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076421.668993:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076421.669005:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076421.669022:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923735296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076421.669033:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923735168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076421.669044:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076421.669051:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076421.669054:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076421.669060:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3110 00000100:00080000:3.0:1769076421.669065:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076421.669069:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076421.669098:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076421.669115:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923735424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076421.669131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076421.669138:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076421.669145:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076421.669148:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076421.669155:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3110 00000100:00080000:3.0:1769076421.669174:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076421.669181:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076421.669190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076421/real 1769076421] req@ffff969e0a7fce00 x1855007923735168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076421.669192:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076421.669218:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923735552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076421.669229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923735168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076421.669239:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076421.669276:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076421.669283:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076421.669291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076421.669293:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3110 00000100:00080000:3.0:1769076421.669307:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076421.669312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076421.669318:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076421.669327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923735168:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076421.669329:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076421.669356:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923735680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076421.669375:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076421.669410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076421.669428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076421/real 1769076421] req@ffff969e0a7fca80 x1855007923735296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076421.669444:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923735296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076421.669468:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076421.669473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076421.669478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076421.669486:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076421.669512:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076421.669524:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076421/real 1769076421] req@ffff969e0a7fdf80 x1855007923735424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076421.669538:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923735424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076421.669548:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076421.669551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076421.669555:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076421.669562:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076421.669580:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076421.669622:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076421/real 1769076421] req@ffff969e0a7fe300 x1855007923735552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076421.669639:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923735552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076421.669649:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076421.669653:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076421.669667:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076421.669689:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076421.669717:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076421.669730:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076421/real 1769076421] req@ffff969e0a7fd180 x1855007923735680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076421.669744:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923735680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076421.669755:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076421.669759:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076421.669762:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076421.988745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076422.692897:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076423.012706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076423.716808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076424.036701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076424.740816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076425.060700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076425.764790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076426.084719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076426.788789:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1769076426.788808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076426.788833:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076426.788842:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076426.788858:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3115 00000100:00080000:2.0:1769076426.788873:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076426.788882:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076426.788941:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923735808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076426.788977:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076426.788982:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076426.788985:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076426.788990:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3115 00000100:00080000:2.0:1769076426.788995:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076426.789001:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076426.789014:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076426.789031:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923735936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076426.789045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923735808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076426.789052:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076426.789058:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076426.789060:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076426.789066:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3115 00000100:00080000:2.0:1769076426.789070:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076426.789073:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076426.789079:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076426.789097:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923736064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076426.789124:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076426.789128:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076426.789131:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076426.789138:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3115 00000100:00080000:2.0:1769076426.789145:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076426.789185:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076426.789194:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076426.789195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076426.789221:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923736192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076426.789236:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076426.789241:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076426.789244:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076426.789249:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3115 00000100:00080000:2.0:1769076426.789256:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076426.789264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076426/real 1769076426] req@ffff969e0a46e300 x1855007923735808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076426.789266:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076426.789277:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076426.789295:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923736320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076426.789325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923735808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076426.789350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076426.789357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076426.789376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923735808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076426.789409:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076426.789445:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076426.789463:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076426/real 1769076426] req@ffff969e0a46ce00 x1855007923735936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076426.789479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923735936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076426.789490:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076426.789495:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076426.789499:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923735936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076426.789506:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076426.789526:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076426.789538:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076426/real 1769076426] req@ffff969e0a46df80 x1855007923736064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076426.789552:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923736064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076426.789561:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076426.789565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076426.789568:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076426.789575:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076426.789629:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076426.789657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076426/real 1769076426] req@ffff969e0a46ca80 x1855007923736192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076426.789673:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923736192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076426.789685:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076426.789689:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076426.789693:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076426.789700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076426.789724:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076426.789736:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076426/real 1769076426] req@ffff969e0a46ea00 x1855007923736320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076426.789750:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923736320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076426.789761:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076426.789764:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076426.789768:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076427.108732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076427.812855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076428.132744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076428.836829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076429.156710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076429.860811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076430.180719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076430.884777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076431.204719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076431.908803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076431.908867:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076431.908882:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076431.908891:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076431.908908:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3121 00000100:00080000:0.0:1769076431.908924:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076431.908933:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076431.908980:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923736448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076431.909017:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076431.909021:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076431.909024:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076431.909029:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3121 00000100:00080000:0.0:1769076431.909034:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076431.909039:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076431.909050:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076431.909068:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923736576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076431.909114:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076431.909119:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076431.909123:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076431.909128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923736448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076431.909133:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3121 00000100:00080000:0.0:1769076431.909161:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076431.909169:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076431.909178:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076431.909198:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923736704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076431.909213:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076431.909217:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076431.909220:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076431.909225:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3121 00000100:00080000:0.0:1769076431.909230:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076431.909235:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076431.909243:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076431.909257:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923736832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076431.909272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736576:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769076431.909277:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076431.909283:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076431.909286:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076431.909300:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3121 00000100:00080000:0.0:1769076431.909305:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076431.909309:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076431.909317:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076431.909338:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923736960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076431.909359:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076431.909415:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076431.909446:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076432.228725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076432.932884:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076433.252684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076433.956823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076434.276706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076434.980917:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076434.981052:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:1.0:1769076434.981108:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076434.981132:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076434.981181:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076434.981192:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076434.981201:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076434.981210:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:3.0:1769076434.981270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076431/real 1769076434] req@ffff969e06a60000 x1855007923736832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076434.981363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923736832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076434.981379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076434.981386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076434.981392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076434.981405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076431/real 1769076434] req@ffff969e06a62d80 x1855007923736960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076434.981430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923736960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076434.981446:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076434.981453:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076434.981460:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076434.981472:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076431/real 1769076434] req@ffff969e06a63800 x1855007923736576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076434.981497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923736576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076434.981514:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076434.981520:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076434.981526:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076434.981538:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076431/real 1769076434] req@ffff969e06a60700 x1855007923736704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076434.981581:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923736704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076434.981658:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076434.981666:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076434.981682:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923736704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076434.981696:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076431/real 1769076434] req@ffff969e06a60a80 x1855007923736448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076434.981726:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923736448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076434.981742:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076434.981749:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076434.981757:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923736448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076435.300684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076436.004809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076436.324704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076437.028732:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076437.028746:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076437.028751:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076437.028764:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3126 00000100:00080000:0.0:1769076437.028779:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076437.028790:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076437.028836:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923737088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076437.028840:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076437.028880:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076437.028887:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076437.028890:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076437.028896:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3126 00000100:00080000:0.0:1769076437.028900:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076437.028906:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076437.028917:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076437.028933:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923737216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076437.028946:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076437.028950:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076437.028952:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076437.028956:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3126 00000100:00080000:0.0:1769076437.028959:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076437.028977:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076437.029013:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076437.029037:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923737344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076437.029054:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076437.029058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923737088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076437.029061:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076437.029067:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076437.029080:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3126 00000100:00080000:0.0:1769076437.029091:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076437.029098:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076437.029109:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076437.029133:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923737472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076437.029151:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076437.029157:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076437.029160:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076437.029167:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3126 00000100:00080000:0.0:1769076437.029175:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076437.029182:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:3.0:1769076437.029195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769076437.029202:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076437.029235:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923737600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076437.029269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076437/real 1769076437] req@ffff969e06a60380 x1855007923737088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076437.029309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923737088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076437.029330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076437.029338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076437.029346:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923737088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076437.029395:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076437.029455:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076437.029483:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076437/real 1769076437] req@ffff969e06a61180 x1855007923737216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076437.029510:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923737216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076437.029526:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076437.029533:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076437.029540:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076437.029581:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076437.029675:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076437.029701:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076437/real 1769076437] req@ffff969e06a61500 x1855007923737344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076437.029724:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923737344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076437.029740:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076437.029746:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076437.029753:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076437.029766:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076437.029807:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076437.029830:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076437/real 1769076437] req@ffff969e06a62680 x1855007923737472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076437.029872:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923737472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076437.029891:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076437.029898:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076437.029904:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076437.030480:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076437.030544:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076437.030571:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076437/real 1769076437] req@ffff969e06a61f80 x1855007923737600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076437.030639:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923737600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076437.030658:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076437.030665:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076437.030672:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076437.348708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076438.052875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076438.372720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076439.076776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076439.396722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076440.100719:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076440.420660:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076441.124815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076441.444729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076442.148766:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076442.148783:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076442.148790:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076442.148808:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3131 00000100:00080000:3.0:1769076442.148828:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076442.148841:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076442.148883:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076442.148945:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923737728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076442.149007:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076442.149012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076442.149014:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076442.149019:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3131 00000100:00080000:3.0:1769076442.149024:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076442.149029:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076442.149044:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076442.149062:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923737856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076442.149074:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076442.149084:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923737728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076442.149085:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076442.149094:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076442.149114:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3131 00000100:00080000:3.0:1769076442.149119:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076442.149123:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076442.149130:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076442.149149:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923737984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076442.149160:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076442.149164:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076442.149166:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076442.149170:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3131 00000100:00080000:3.0:1769076442.149188:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076442.149193:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076442.149203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769076442.149207:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076442.149231:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923738112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076442.149245:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076442.149251:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076442.149254:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076442.149264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076442/real 1769076442] req@ffff969e0a7fc380 x1855007923737728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076442.149264:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3131 00000100:00080000:3.0:1769076442.149284:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076442.149303:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076442.149318:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076442.149329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923737728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076442.149335:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923738240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076442.149370:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076442.149379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076442.149387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923737728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076442.149424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076442.149456:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076442.149471:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076442/real 1769076442] req@ffff969e0a7fc000 x1855007923737856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076442.149487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923737856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076442.149498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076442.149502:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076442.149506:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076442.149513:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076442.149532:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076442.149543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076442/real 1769076442] req@ffff969e0a7fd500 x1855007923737984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076442.149557:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923737984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076442.149567:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076442.149570:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076442.149574:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923737984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076442.149626:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076442.149657:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076442.149671:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076442/real 1769076442] req@ffff969e0a7fdc00 x1855007923738112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076442.149687:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923738112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076442.149705:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076442.149711:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076442.149718:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076442.149730:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076442.149769:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076442.149791:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076442/real 1769076442] req@ffff969e0a7ff800 x1855007923738240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076442.149817:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923738240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076442.149828:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076442.149832:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076442.149837:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076442.468736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076443.172967:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076443.492658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076444.196719:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076444.516662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076445.220769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076445.540679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076446.244887:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076446.564698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076447.268788:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076447.268810:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076447.268850:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076447.268883:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3136 00010000:00010000:0.0:1769076447.268899:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076447.268906:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076447.268921:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076447.268998:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923738368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076447.269070:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076447.269091:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076447.269096:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076447.269106:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3136 00000100:00080000:2.0:1769076447.269115:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076447.269124:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076447.269138:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923738368:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076447.269144:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076447.269198:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923738496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076447.269246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076447.269283:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076447.269291:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076447.269296:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076447.269305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076447/real 1769076447] req@ffff969e0a46d880 x1855007923738368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076447.269310:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3136 00000100:00080000:2.0:1769076447.269337:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076447.269349:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923738368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076447.269353:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076447.269377:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769076447.269381:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076447.269393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076447.269399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923738368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076447.269408:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923738624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076447.269423:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076447.269428:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076447.269432:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076447.269439:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738496:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076447.269449:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3136 00000100:00080000:2.0:1769076447.269462:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076447.269470:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076447.269476:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076447.269491:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076447.269499:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923738752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076447.269516:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076447.269531:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076447.269540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076447/real 1769076447] req@ffff969e0a46ed80 x1855007923738496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076447.269541:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076447.269565:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3136 00000100:00080000:2.0:1769076447.269584:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076447.269585:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923738496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076447.269659:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076447.269667:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076447.269682:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076447.269684:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738496:192.168.204.143@tcp:38:kworker.0 00000008:00100000:2.0:1769076447.269707:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076447.269716:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076447.269737:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923738880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076447.269755:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076447.269783:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076447/real 1769076447] req@ffff969e0a46f480 x1855007923738624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076447.269836:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923738624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076447.269847:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076447.269851:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076447.269855:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076447.269877:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076447.269903:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076447.269916:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076447/real 1769076447] req@ffff969e0a46d500 x1855007923738752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076447.269928:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923738752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076447.269937:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076447.269941:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076447.269944:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076447.269950:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076447.269966:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076447.269976:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076447/real 1769076447] req@ffff969e0a46c380 x1855007923738880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076447.269987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923738880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076447.270012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076447.270017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076447.270020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923738880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076447.588668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076448.292720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076448.612714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076449.316735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076449.636721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076450.340738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076450.660720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076451.364797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076451.684691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076452.388795:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076452.388810:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076452.388816:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076452.388829:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3141 00000100:00080000:2.0:1769076452.388844:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076452.388853:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076452.388896:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076452.388901:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923739008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076452.388941:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076452.388946:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076452.388948:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076452.388953:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3141 00000100:00080000:2.0:1769076452.388958:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076452.388964:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076452.388975:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076452.388996:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923739136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076452.388998:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923739008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076452.389023:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076452.389047:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076452.389053:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076452.389059:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3141 00000100:00080000:2.0:1769076452.389065:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076452.389071:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076452.389079:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076452.389100:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923739264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076452.389116:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076452.389121:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076452.389123:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076452.389129:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3141 00000100:00080000:2.0:1769076452.389134:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076452.389137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076452.389145:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076452.389155:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076452.389173:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923739392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076452.389187:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076452.389192:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076452.389200:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076452.389201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076452/real 1769076452] req@ffff969e0a46f800 x1855007923739008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076452.389214:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3141 00000100:00080000:2.0:1769076452.389225:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076452.389235:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076452.389250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923739008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076452.389255:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076452.389280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769076452.389283:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923739520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076452.389291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076452.389307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923739008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076452.389350:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076452.389378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076452.389394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076452/real 1769076452] req@ffff969e0a46f100 x1855007923739136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076452.389409:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923739136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076452.389436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076452.389441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076452.389446:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076452.389453:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076452.389479:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076452.389492:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076452/real 1769076452] req@ffff969e0a46c380 x1855007923739264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076452.389506:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923739264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076452.389516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076452.389520:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076452.389523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076452.389530:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076452.389549:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076452.389560:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076452/real 1769076452] req@ffff969e0a46d500 x1855007923739392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076452.389574:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923739392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076452.389584:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076452.389587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076452.389636:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076452.389645:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076452.389674:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076452.389689:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076452/real 1769076452] req@ffff969e0a46f480 x1855007923739520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076452.389703:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923739520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076452.389714:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076452.389718:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076452.389722:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076452.708748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076453.412975:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076453.732666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076454.436823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076454.756734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076455.460780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076455.780726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076456.484817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076456.804725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076457.508811:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076457.508825:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076457.508831:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076457.508844:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3146 00000100:00080000:0.0:1769076457.508859:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076457.508870:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076457.508882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076457.508919:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923739648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076457.508981:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076457.508986:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076457.508989:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076457.508994:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3146 00000100:00080000:0.0:1769076457.508999:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076457.509004:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076457.509017:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076457.509033:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923739776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076457.509049:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923739648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076457.509085:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076457.509090:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076457.509093:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076457.509101:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3146 00000100:00080000:0.0:1769076457.509122:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076457.509133:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076457.509142:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076457.509154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076457.509167:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923739904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076457.509184:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076457.509189:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076457.509192:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076457.509197:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3146 00000100:00080000:0.0:1769076457.509202:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076457.509208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076457/real 1769076457] req@ffff969e06a63100 x1855007923739648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076457.509214:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076457.509227:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076457.509251:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923740032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076457.509259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923739648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076457.509294:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076457.509301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076457.509302:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076457.509312:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076457.509318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076457.509324:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3146 00000100:00100000:1.0:1769076457.509344:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923739648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076457.509347:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076457.509363:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076457.509370:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076457.509386:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923740160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076457.509394:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076457.509457:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076457.509477:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076457/real 1769076457] req@ffff969e06a61880 x1855007923739776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076457.509493:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923739776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076457.509504:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076457.509508:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076457.509513:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076457.509520:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076457.509539:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076457.509550:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076457/real 1769076457] req@ffff969e06a61f80 x1855007923739904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076457.509563:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923739904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076457.509573:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076457.509579:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076457.509583:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923739904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076457.509634:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076457.509700:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076457.509741:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076457/real 1769076457] req@ffff969e06a62680 x1855007923740032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076457.509768:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923740032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076457.509779:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076457.509783:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076457.509787:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076457.509811:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076457.509840:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076457.509853:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076457/real 1769076457] req@ffff969e06a61500 x1855007923740160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076457.509867:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923740160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076457.509877:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076457.509880:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076457.509884:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076457.828757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076458.532879:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076458.852711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076459.557267:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076459.876799:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076460.581260:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076460.900725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076461.605678:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076461.924742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076462.628971:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076462.628989:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076462.628996:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076462.629011:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3151 00000100:00080000:0.0:1769076462.629028:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076462.629039:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076462.629097:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923740288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076462.629143:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076462.629151:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076462.629156:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076462.629165:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3151 00000100:00080000:0.0:1769076462.629173:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076462.629181:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076462.629196:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076462.629223:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923740416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076462.629241:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076462.629262:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076462.629266:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076462.629273:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3151 00000100:00080000:0.0:1769076462.629281:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076462.629287:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076462.629299:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076462.629310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923740288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076462.629330:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923740544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076462.629332:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076462.629359:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076462.629367:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076462.629371:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076462.629420:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3151 00000100:00080000:0.0:1769076462.629429:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076462.629436:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076462.629446:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076462.629454:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076462.629474:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923740672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076462.629494:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076462.629500:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076462.629503:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076462.629510:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3151 00000100:00080000:0.0:1769076462.629518:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076462.629525:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076462.629532:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076462/real 1769076462] req@ffff969e06a61180 x1855007923740288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076462.629550:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076462.629635:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923740288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076462.629668:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076462.629679:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923740800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076462.629683:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076462.629700:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923740288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076462.629726:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076462.629775:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076462.629799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076462/real 1769076462] req@ffff969e06a60380 x1855007923740416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076462.629823:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923740416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076462.629838:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076462.629844:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076462.629850:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076462.629887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076462.629926:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076462.629946:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076462/real 1769076462] req@ffff969e06a60a80 x1855007923740544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076462.629969:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923740544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076462.630015:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076462.630022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076462.630028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076462.630036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076462.630070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076462.630090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076462/real 1769076462] req@ffff969e06a60700 x1855007923740672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076462.630111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923740672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076462.630127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076462.630134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076462.630140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076462.630151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076462.630183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076462.630197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076462/real 1769076462] req@ffff969e06a63800 x1855007923740800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076462.630213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923740800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076462.630228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076462.630233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076462.630251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923740800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076462.948713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076463.653354:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076463.972730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076464.677116:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076464.996750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076465.700897:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076466.020733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076466.724784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076467.044695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076467.748741:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076467.748753:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076467.748758:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076467.748770:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3156 00000100:00080000:1.0:1769076467.748783:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076467.748793:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076467.748836:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923740928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076467.748877:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076467.748882:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076467.748885:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076467.748890:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3156 00000100:00080000:1.0:1769076467.748894:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076467.748899:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076467.748911:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076467.748927:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923741056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076467.748942:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076467.748948:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923740928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769076467.748949:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076467.748989:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076467.748998:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3156 00000100:00080000:1.0:1769076467.749007:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076467.749013:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076467.749020:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076467.749044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076467.749046:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923741184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076467.749064:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076467.749069:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076467.749072:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076467.749078:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3156 00000100:00080000:1.0:1769076467.749084:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076467.749090:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076467.749097:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076467.749108:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923741312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076467.749121:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076467/real 1769076467] req@ffff969e10b60700 x1855007923740928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076467.749127:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076467.749145:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076467.749152:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076467.749168:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3156 00000100:00080000:1.0:1769076467.749174:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076467.749185:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076467.749186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923740928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076467.749191:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769076467.749204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076467.749214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076467.749222:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923741440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076467.749225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923740928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076467.749282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076467.749322:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076467.749343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076467/real 1769076467] req@ffff969e10b62d80 x1855007923741056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076467.749360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923741056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076467.749371:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076467.749376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076467.749381:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076467.749391:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076467.749416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00010000:00010000:0.0:1769076467.749420:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076467.749433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076467/real 1769076467] req@ffff969e10b60000 x1855007923741184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076467.749449:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923741184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076467.749461:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076467.749465:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076467.749495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076467.749504:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076467.749527:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076467.749540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076467/real 1769076467] req@ffff969e10b61180 x1855007923741312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076467.749567:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923741312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076467.749578:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076467.749582:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076467.749586:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076467.749631:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076467.749662:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076467.749676:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076467/real 1769076467] req@ffff969e10b60380 x1855007923741440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076467.749691:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923741440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076467.749702:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076467.749706:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076467.749710:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076468.068697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076468.772812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076469.092694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076469.796915:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076470.116787:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076470.820871:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076471.140793:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076471.844964:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076472.164791:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076472.868805:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076472.868824:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076472.868831:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076472.868845:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3162 00000100:00080000:1.0:1769076472.868865:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076472.868877:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076472.868943:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923741568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076472.868988:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076472.868993:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076472.868996:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076472.869003:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3162 00000100:00080000:1.0:1769076472.869008:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076472.869013:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076472.869028:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076472.869047:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923741696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076472.869061:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076472.869098:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076472.869103:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076472.869114:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3162 00000100:00080000:1.0:1769076472.869122:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076472.869135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923741568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769076472.869143:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076472.869163:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076472.869200:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923741824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076472.869221:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076472.869227:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076472.869231:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076472.869239:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3162 00000100:00080000:1.0:1769076472.869247:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076472.869260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769076472.869267:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076472.869278:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076472.869298:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923741952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076472.869313:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076472.869318:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076472.869322:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076472.869332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076472/real 1769076472] req@ffff969e10b60380 x1855007923741568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076472.869333:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3162 00000100:00080000:1.0:1769076472.869347:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076472.869368:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076472.869416:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923741568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076472.869423:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076472.869459:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923742080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076472.869537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076472.869546:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076472.869556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923741568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076472.869575:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076472.869696:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00010000:00010000:0.0:1769076472.869704:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076472.869720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076472/real 1769076472] req@ffff969e10b61180 x1855007923741696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076472.869741:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923741696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076472.869755:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076472.869761:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076472.869766:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076472.869799:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076472.869821:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076472.869833:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076472/real 1769076472] req@ffff969e10b60000 x1855007923741824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076472.869847:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923741824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076472.869857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076472.869861:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076472.869865:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076472.869888:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076472.869915:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076472.869928:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076472/real 1769076472] req@ffff969e10b62d80 x1855007923741952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076472.869943:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923741952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076472.869953:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076472.869957:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076472.869961:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923741952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076472.869968:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076472.869987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076472.869997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076472/real 1769076472] req@ffff969e10b60700 x1855007923742080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076472.870011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923742080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076472.870021:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076472.870024:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076472.870028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076473.188687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076473.892901:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076474.212714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076474.916889:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076475.236726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076475.940865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076476.260774:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076476.964857:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076477.284725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076477.988733:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076477.988746:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076477.988752:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076477.988764:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3167 00000100:00080000:1.0:1769076477.988778:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076477.988787:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076477.988833:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923742208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076477.988875:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076477.988880:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076477.988882:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076477.988888:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3167 00000100:00080000:1.0:1769076477.988892:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076477.988897:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076477.988909:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076477.988923:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923742336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076477.988977:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076477.988982:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076477.988984:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076477.988991:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3167 00000100:00080000:1.0:1769076477.988996:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076477.989002:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076477.989010:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076477.989020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923742208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076477.989033:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923742464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076477.989048:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076477.989054:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076477.989056:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076477.989061:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3167 00000100:00080000:1.0:1769076477.989065:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076477.989070:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076477.989076:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076477.989090:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923742592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076477.989101:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076477.989105:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076477.989107:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076477.989118:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3167 00000800:00000100:3.0:1769076477.989120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769076477.989136:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076477.989144:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076477.989153:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076477.989172:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923742720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076477.989179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076477/real 1769076477] req@ffff969e10b62680 x1855007923742208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076477.989226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923742208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076477.989241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076477.989249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076477.989256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923742208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076477.989270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076477.989331:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076477.989349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076477/real 1769076477] req@ffff969e10b62a00 x1855007923742336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076477.989367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923742336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076477.989379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076477.989385:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00010000:00010000:0.0:1769076477.989390:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076477.989392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076477.989430:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076477.989454:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076477.989466:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076477/real 1769076477] req@ffff969e10b60a80 x1855007923742464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076477.989480:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923742464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076477.989508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076477.989512:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076477.989517:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076477.989524:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076477.989549:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076477.989562:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076477/real 1769076477] req@ffff969e10b61f80 x1855007923742592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076477.989576:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923742592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076477.989586:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076477.989631:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076477.989636:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076477.989644:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076477.989668:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076477.989682:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076477/real 1769076477] req@ffff969e10b61500 x1855007923742720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076477.989696:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923742720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076477.989707:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076477.989711:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076477.989724:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076478.308694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076479.012834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076479.332716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076480.036873:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076480.356707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076481.060919:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076481.380721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076482.084906:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076482.404773:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076483.108801:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076483.108820:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076483.108829:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076483.108847:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3172 00010000:00010000:2.0:1769076483.108858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076483.108866:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076483.108882:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076483.108941:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923742848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076483.109022:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076483.109027:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076483.109030:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076483.109038:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3172 00000100:00080000:0.0:1769076483.109048:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076483.109057:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076483.109074:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076483.109106:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923742976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076483.109127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923742848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076483.109140:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076483.109151:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076483.109155:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076483.109165:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3172 00000100:00080000:0.0:1769076483.109176:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076483.109183:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076483.109192:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076483.109245:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923743104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076483.109260:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076483.109265:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076483.109269:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076483.109275:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3172 00000100:00080000:0.0:1769076483.109293:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076483.109304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076483.109311:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076483.109324:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076483.109345:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923743232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076483.109359:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076483.109364:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076483.109368:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076483.109376:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3172 00000100:00080000:0.0:1769076483.109381:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076483.109394:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076483.109396:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076483/real 1769076483] req@ffff969e06a62d80 x1855007923742848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076483.109403:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076483.109434:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923743360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076483.109450:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923742848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076483.109484:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076483.109495:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076483.109503:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923742848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076483.109543:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076483.109579:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076483.109643:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076483/real 1769076483] req@ffff969e06a60000 x1855007923742976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076483.109661:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923742976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076483.109673:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076483.109678:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076483.109682:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923742976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076483.109691:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076483.109717:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076483.109753:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076483/real 1769076483] req@ffff969e06a63800 x1855007923743104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076483.109769:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923743104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076483.109780:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076483.109785:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076483.109789:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076483.109797:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076483.109828:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076483.109849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076483/real 1769076483] req@ffff969e06a60700 x1855007923743232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076483.109873:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923743232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076483.109891:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076483.109898:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076483.109906:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076483.109919:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076483.109954:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076483.109970:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076483/real 1769076483] req@ffff969e06a60a80 x1855007923743360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076483.109986:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923743360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076483.110007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076483.110012:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076483.110016:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076483.428692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076484.132923:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076484.452682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076485.156920:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076485.476718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076486.180878:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076486.500700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076487.204845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076487.524698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076488.228777:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076488.228794:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076488.228800:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076488.228813:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3177 00000100:00080000:3.0:1769076488.228827:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076488.228837:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076488.228886:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923743488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076488.228922:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076488.228945:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076488.228955:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076488.228961:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076488.228979:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3177 00000100:00080000:3.0:1769076488.228989:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076488.228999:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076488.229016:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076488.229023:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923743488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076488.229048:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923743616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076488.229066:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076488.229071:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076488.229094:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076488.229100:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3177 00000100:00080000:3.0:1769076488.229108:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076488.229114:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076488.229122:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076488.229171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076488.229190:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923743744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076488.229207:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076488.229212:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076488.229215:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076488.229222:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3177 00000100:00100000:1.0:1769076488.229229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076488/real 1769076488] req@ffff969e0a7ffb80 x1855007923743488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076488.229233:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076488.229247:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076488.229261:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076488.229272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923743488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076488.229281:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923743872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076488.229314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076488.229316:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076488.229324:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076488.229327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076488.229329:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076488.229342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923743488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076488.229343:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3177 00000100:00080000:3.0:1769076488.229353:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076488.229360:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076488.229366:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076488.229386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076488.229396:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923744000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076488.229462:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076488.229480:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076488/real 1769076488] req@ffff969e0a7ff800 x1855007923743616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076488.229497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923743616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076488.229508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076488.229512:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076488.229517:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076488.229524:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076488.229543:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076488.229554:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076488/real 1769076488] req@ffff969e0a7fdc00 x1855007923743744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076488.229568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923743744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076488.229578:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076488.229581:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076488.229585:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076488.229626:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076488.229649:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076488.229662:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076488/real 1769076488] req@ffff969e0a7fd500 x1855007923743872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076488.229697:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923743872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076488.229709:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076488.229713:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076488.229717:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923743872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076488.229739:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076488.229762:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076488.229774:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076488/real 1769076488] req@ffff969e0a7fc000 x1855007923744000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076488.229788:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923744000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076488.229798:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076488.229801:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076488.229805:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076488.548741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076489.252875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076489.572713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076490.276829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076490.596759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076491.300816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076491.620734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076492.324856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076492.644735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076493.348849:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076493.348863:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076493.348869:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076493.348883:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3182 00000100:00080000:0.0:1769076493.348898:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769076493.348906:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076493.348908:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076493.348963:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923744128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076493.349003:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076493.349031:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076493.349034:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076493.349090:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3182 00000100:00080000:0.0:1769076493.349097:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076493.349104:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076493.349115:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076493.349117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923744128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076493.349161:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923744256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076493.349175:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076493.349178:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076493.349181:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076493.349185:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3182 00000100:00080000:0.0:1769076493.349200:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076493.349204:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076493.349210:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076493.349217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076493.349229:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923744384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076493.349241:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076493.349244:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076493.349247:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076493.349250:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3182 00000100:00080000:0.0:1769076493.349254:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076493.349258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076493/real 1769076493] req@ffff969e06a60380 x1855007923744128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076493.349261:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076493.349281:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076493.349299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923744128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076493.349300:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923744512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076493.349349:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076493.349351:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076493.349355:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076493.349359:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076493.349360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076493.349367:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3182 00000100:00100000:1.0:1769076493.349372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923744128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076493.349375:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076493.349384:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076493.349389:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076493.349398:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923744640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076493.349407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076493.349437:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076493.349449:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076493/real 1769076493] req@ffff969e06a61180 x1855007923744256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076493.349461:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923744256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076493.349469:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076493.349472:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076493.349475:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076493.349480:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076493.349493:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076493.349501:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076493/real 1769076493] req@ffff969e06a61500 x1855007923744384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076493.349529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923744384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076493.349557:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076493.349561:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076493.349566:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076493.349573:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076493.349598:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076493.349611:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076493/real 1769076493] req@ffff969e06a62680 x1855007923744512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076493.349656:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923744512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076493.349667:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076493.349672:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076493.349676:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076493.349698:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076493.349721:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076493.349734:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076493/real 1769076493] req@ffff969e06a61f80 x1855007923744640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076493.349749:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923744640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076493.349760:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076493.349763:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076493.349777:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076493.668706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076494.372896:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076494.692714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076495.396916:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076495.716683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076496.420789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076496.740692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076497.444827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076497.764701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076498.468696:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076498.468709:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076498.468714:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076498.468727:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3187 00000100:00080000:3.0:1769076498.468741:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076498.468748:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076498.468796:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923744768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076498.468797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076498.468828:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076498.468831:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076498.468833:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076498.468837:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3187 00000100:00080000:3.0:1769076498.468841:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076498.468846:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076498.468856:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076498.468872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923744768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076498.468872:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923744896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076498.468907:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076498.468911:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076498.468913:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076498.468917:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3187 00000100:00080000:3.0:1769076498.468920:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076498.468944:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076498.468949:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076498.468974:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923745024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076498.468982:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076498.468985:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076498.468987:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076498.468994:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3187 00000100:00080000:3.0:1769076498.469000:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076498.469006:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076498.469013:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076498.469019:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076498.469026:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923745152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076498.469037:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076498.469041:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076498.469043:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076498.469047:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3187 00000100:00080000:3.0:1769076498.469050:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076498.469054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745024:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076498.469061:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076498.469068:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076498.469084:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923745280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076498.469088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076498.469132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076498.788665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076499.492787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076499.812718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076500.516722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076500.836670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076501.540769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076501.540899:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:1.0:1769076501.540977:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076501.540997:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076501.541060:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076501.541068:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076501.541077:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:1.0:1769076501.541105:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:3.0:1769076501.541131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076498/real 1769076501] req@ffff969e0a7fdf80 x1855007923745152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.541190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923745152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.541210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.541217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076501.541222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076501.541231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076498/real 1769076501] req@ffff969e0a7fd180 x1855007923744896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.541246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923744896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.541256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.541260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076501.541264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923744896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076501.541272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076498/real 1769076501] req@ffff969e0a7fe300 x1855007923745024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.541286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923745024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.541295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.541299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076501.541303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076501.541311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076498/real 1769076501] req@ffff969e0a7fc380 x1855007923744768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.541338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923744768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.541349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.541353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076501.541358:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923744768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076501.541390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076498/real 1769076501] req@ffff969e0a7fca80 x1855007923745280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076501.541406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923745280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076501.541417:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076501.541422:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076501.541427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076501.860671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076502.564728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076502.884754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076503.588723:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076503.588734:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076503.588738:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076503.588749:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3192 00010000:00010000:2.0:1769076503.588758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076503.588760:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076503.588767:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076503.588802:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923745408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076503.588826:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076503.588843:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076503.588845:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076503.588848:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3192 00000100:00080000:3.0:1769076503.588852:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076503.588858:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923745408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076503.588859:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076503.588887:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076503.588906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923745536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076503.588920:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076503.588923:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076503.588925:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076503.588929:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3192 00000100:00080000:3.0:1769076503.588933:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076503.588937:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076503.588944:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076503.588957:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923745664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076503.588966:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076503.588973:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076503.588973:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076503.588978:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076503.588984:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3192 00000100:00080000:3.0:1769076503.588989:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076503.588994:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076503.589007:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076503.589014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076503/real 1769076503] req@ffff969e0a7fca80 x1855007923745408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076503.589022:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923745792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076503.589039:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076503.589044:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076503.589045:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923745408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076503.589048:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076503.589055:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3192 00000100:00080000:3.0:1769076503.589063:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076503.589069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076503.589070:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076503.589080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:3.0:1769076503.589080:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076503.589095:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923745920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076503.589098:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923745408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076503.589133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076503.589161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076503.589178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076503/real 1769076503] req@ffff969e0a7fc380 x1855007923745536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076503.589192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923745536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076503.589201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076503.589205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076503.589208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076503.589215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076503.589237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076503.589247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076503/real 1769076503] req@ffff969e0a7fe300 x1855007923745664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076503.589273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923745664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076503.589281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076503.589285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076503.589288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076503.589294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076503.589321:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076503.589332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076503/real 1769076503] req@ffff969e0a7fd180 x1855007923745792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076503.589347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923745792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076503.589357:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076503.589360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076503.589363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076503.589369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076503.589387:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076503.589397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076503/real 1769076503] req@ffff969e0a7fdf80 x1855007923745920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076503.589409:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923745920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076503.589436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076503.589441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076503.589445:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923745920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076503.908662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076504.612776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076504.932667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076505.636692:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076505.956669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076506.660667:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076506.980658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076507.684679:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076508.004677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076508.708677:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076508.708687:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076508.708692:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769076508.708697:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076508.708703:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3197 00000100:00080000:3.0:1769076508.708715:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076508.708722:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076508.708758:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923746048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076508.708794:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076508.708798:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076508.708801:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076508.708807:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3197 00000100:00100000:1.0:1769076508.708822:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923746048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076508.708827:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076508.708835:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076508.708845:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076508.708860:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923746176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076508.708874:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076508.708878:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1769076508.708880:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076508.708882:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076508.708890:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3197 00000100:00080000:3.0:1769076508.708895:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076508.708901:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076508.708907:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076508.708913:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076508/real 1769076508] req@ffff969e0a7fce00 x1855007923746048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076508.708924:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923746304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076508.708931:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923746048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076508.708944:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076508.708947:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076508.708953:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076508.708955:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076508.708957:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076508.708960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923746048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076508.708967:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3197 00000100:00080000:3.0:1769076508.708973:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076508.708978:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076508.708982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746176:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769076508.708986:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076508.709004:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923746432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076508.709005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076508.709027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076508/real 1769076508] req@ffff969e0a7fea00 x1855007923746176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076508.709035:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076508.709042:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076508.709044:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076508.709048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923746176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076508.709052:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3197 00000100:00080000:3.0:1769076508.709061:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076508.709066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076508.709068:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076508.709075:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000008:00100000:3.0:1769076508.709077:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076508.709092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076508.709093:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923746560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076508.709100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076508.709127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076508.709137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076508/real 1769076508] req@ffff969e0a7fc700 x1855007923746304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076508.709146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923746304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076508.709151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076508.709153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076508.709155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076508.709165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076508.709176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076508.709182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076508/real 1769076508] req@ffff969e0a7fd880 x1855007923746432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076508.709189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923746432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076508.709199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076508.709202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076508.709204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076508.709208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076508.709220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076508.709226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076508/real 1769076508] req@ffff969e0a7fe680 x1855007923746560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076508.709233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923746560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076508.709238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076508.709240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076508.709241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076509.028730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076509.732756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076510.052674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076510.756730:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076511.076705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076511.780719:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076512.100801:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076512.804735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076513.124774:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076513.828676:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076513.828685:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076513.828689:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076513.828712:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3203 00010000:00010000:1.0:1769076513.828716:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076513.828723:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076513.828730:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076513.828763:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923746688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076513.828784:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076513.828786:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076513.828788:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076513.828791:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3203 00000100:00080000:3.0:1769076513.828793:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076513.828795:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076513.828817:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923746688:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076513.828817:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076513.828833:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923746816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076513.828842:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076513.828844:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076513.828846:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076513.828849:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3203 00000100:00080000:3.0:1769076513.828852:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076513.828855:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076513.828858:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076513.828866:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923746944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076513.828873:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076513.828876:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076513.828877:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076513.828879:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3203 00000100:00080000:3.0:1769076513.828881:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076513.828886:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076513.828887:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769076513.828890:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076513.828899:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923747072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076513.828905:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076513.828908:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076513.828914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076513.828919:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076513/real 1769076513] req@ffff969e0a7fed80 x1855007923746688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076513.828921:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3203 00000100:00080000:3.0:1769076513.828927:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076513.828933:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076513.828939:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076513.828945:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923746688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076513.828950:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923747200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076513.828964:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076513.828970:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076513.828973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923746688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076513.828993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076513.829007:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076513.829015:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076513/real 1769076513] req@ffff969e0a7fe680 x1855007923746816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076513.829023:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923746816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076513.829029:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076513.829031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076513.829033:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076513.829036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076513.829053:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076513.829063:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076513/real 1769076513] req@ffff969e0a7fd880 x1855007923746944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076513.829091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923746944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076513.829100:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076513.829104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076513.829106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923746944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076513.829111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076513.829128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076513.829136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076513/real 1769076513] req@ffff969e0a7fc700 x1855007923747072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076513.829144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923747072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076513.829149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076513.829152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076513.829153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076513.829157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076513.829166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076513.829171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076513/real 1769076513] req@ffff969e0a7fea00 x1855007923747200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076513.829178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923747200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076513.829193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076513.829195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076513.829197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076514.148733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076514.852933:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076515.172668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076515.876744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076516.196653:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076516.900723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076517.220677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076517.924761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076518.244670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076518.948775:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076518.948787:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076518.948792:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076518.948803:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3208 00000100:00080000:1.0:1769076518.948818:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076518.948829:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076518.948849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076518.948887:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923747328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076518.948924:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076518.948928:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076518.948930:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076518.948936:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3208 00000100:00080000:1.0:1769076518.948940:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076518.948968:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076518.948979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923747328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:1.0:1769076518.948981:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076518.949001:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923747456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076518.949009:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076518.949012:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076518.949014:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076518.949017:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3208 00000100:00080000:1.0:1769076518.949021:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076518.949024:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076518.949028:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076518.949036:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923747584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076518.949047:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076518.949051:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076518.949053:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769076518.949058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769076518.949062:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3208 00000100:00080000:1.0:1769076518.949068:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076518.949073:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076518.949080:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076518.949094:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923747712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076518.949098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076518/real 1769076518] req@ffff969e10b61500 x1855007923747328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076518.949116:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076518.949125:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769076518.949128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923747328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076518.949129:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076518.949137:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3208 00000100:00080000:1.0:1769076518.949143:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076518.949148:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:3.0:1769076518.949149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1769076518.949152:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769076518.949159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076518.949164:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923747840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076518.949168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923747328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076518.949211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076518.949236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076518.949250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076518/real 1769076518] req@ffff969e10b61f80 x1855007923747456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076518.949265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923747456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076518.949301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076518.949304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076518.949307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076518.949312:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076518.949329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076518.949337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076518/real 1769076518] req@ffff969e10b60a80 x1855007923747584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076518.949345:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923747584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076518.949351:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076518.949354:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076518.949356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076518.949359:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076518.949371:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076518.949376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076518/real 1769076518] req@ffff969e10b62a00 x1855007923747712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076518.949383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923747712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076518.949390:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076518.949392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076518.949398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076518.949403:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076518.949418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076518.949424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076518/real 1769076518] req@ffff969e10b62680 x1855007923747840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076518.949432:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923747840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076518.949438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076518.949440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076518.949442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923747840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076519.268697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076519.972969:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076520.292737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076520.996860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076521.316719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076522.020866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076522.340721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076523.044817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076523.364717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076524.068724:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076524.068738:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076524.068744:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076524.068757:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3213 00000100:00080000:1.0:1769076524.068770:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076524.068779:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076524.068827:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923747968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076524.068867:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076524.068872:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076524.068874:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076524.068879:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3213 00000100:00080000:1.0:1769076524.068906:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076524.068914:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076524.068927:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076524.068951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923747968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076524.068952:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923748096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076524.068980:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076524.068985:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076524.068988:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076524.068995:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3213 00000100:00080000:1.0:1769076524.068999:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076524.069004:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076524.069011:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076524.069026:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923748224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076524.069039:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076524.069043:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076524.069046:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769076524.069055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769076524.069063:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3213 00000100:00080000:1.0:1769076524.069072:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076524.069078:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076524.069085:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076524.069101:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923748352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076524.069132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076524/real 1769076524] req@ffff969e10b60700 x1855007923747968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076524.069141:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076524.069157:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076524.069163:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076524.069175:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3213 00000100:00080000:1.0:1769076524.069183:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076524.069188:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076524.069197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923747968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076524.069199:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076524.069222:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923748480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076524.069231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076524.069248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076524.069260:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923747968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076524.069315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076524.069356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00010000:00010000:0.0:1769076524.069390:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076524.069395:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076524/real 1769076524] req@ffff969e10b62d80 x1855007923748096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076524.069416:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923748096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076524.069430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076524.069435:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076524.069440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076524.069448:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769076524.069471:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076524.069483:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076524/real 1769076524] req@ffff969e10b60000 x1855007923748224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076524.069506:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923748224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076524.069522:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076524.069555:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076524.069563:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076524.069578:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076524.069665:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076524.069682:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076524/real 1769076524] req@ffff969e10b61180 x1855007923748352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076524.069707:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923748352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076524.069731:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076524.069736:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076524.069740:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076524.069748:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769076524.069800:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076524.069815:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076524/real 1769076524] req@ffff969e10b60380 x1855007923748480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076524.069829:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923748480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076524.069840:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076524.069844:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769076524.069847:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076524.388701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076525.092913:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076525.412732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076526.117028:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076526.436754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076527.141294:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076527.460777:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076528.164762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076528.484674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076529.188862:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076529.188882:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076529.188901:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076529.188920:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3218 00000100:00080000:3.0:1769076529.188938:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076529.188950:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076529.189009:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923748608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076529.189014:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076529.189062:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076529.189093:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076529.189099:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076529.189110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923748608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076529.189118:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3218 00000100:00080000:3.0:1769076529.189137:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076529.189149:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076529.189168:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076529.189361:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076529.189420:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076529/real 1769076529] req@ffff969e0a7fce00 x1855007923748608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076529.189471:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923748608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076529.189490:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076529.189497:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076529.189504:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923748608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076529.189859:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923748736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076529.189903:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076529.189910:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076529.189914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076529.189922:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3218 00000100:00080000:3.0:1769076529.189929:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076529.189936:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076529.189948:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076529.189975:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923748864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076529.189991:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076529.189993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748736:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076529.189997:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076529.190005:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076529.190012:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3218 00000100:00080000:3.0:1769076529.190019:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076529.190024:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076529.190042:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769076529.190050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076529.190066:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923748992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076529.190079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076529/real 1769076529] req@ffff969e0a7fdf80 x1855007923748736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076529.190085:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076529.190099:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076529.190105:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076529.190120:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3218 00000100:00100000:2.0:1769076529.190128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923748736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076529.190133:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076529.190150:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769076529.190151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076529.190164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000008:00100000:3.0:1769076529.190165:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076529.190171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076529.190197:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923749120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076529.190199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076529.190229:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076529.190250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076529/real 1769076529] req@ffff969e0a7fd180 x1855007923748864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076529.190295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923748864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076529.190305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076529.190309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076529.190313:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076529.190320:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076529.190346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076529.190358:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076529/real 1769076529] req@ffff969e0a7fe300 x1855007923748992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076529.190370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923748992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076529.190379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076529.190383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076529.190386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923748992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076529.190402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076529.190429:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076529.190472:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076529/real 1769076529] req@ffff969e0a7fc380 x1855007923749120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076529.190495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923749120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076529.190546:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076529.190551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076529.190557:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076529.508753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076530.213668:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076530.532722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076531.237231:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076531.556808:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076532.260908:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076532.580747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076533.284881:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076533.604705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076534.308708:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076534.308720:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076534.308726:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076534.308737:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3223 00000100:00080000:3.0:1769076534.308749:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076534.308756:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076534.308802:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923749248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076534.308818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076534.308835:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076534.308840:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076534.308841:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076534.308845:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3223 00000100:00080000:3.0:1769076534.308866:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076534.308870:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076534.308882:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076534.308897:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923749376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076534.308914:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076534.308915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923749248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076534.308924:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076534.308929:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076534.308935:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3223 00000100:00080000:3.0:1769076534.308939:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076534.308944:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076534.308949:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076534.308965:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923749504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076534.308973:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076534.308976:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076534.308977:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076534.308982:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3223 00000100:00080000:3.0:1769076534.308987:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076534.308990:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076534.308994:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769076534.309000:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076534.309011:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923749632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076534.309031:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076534.309034:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076534.309036:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076534.309042:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076534/real 1769076534] req@ffff969e0a7fca80 x1855007923749248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076534.309047:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3223 00000100:00080000:3.0:1769076534.309062:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076534.309066:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076534.309071:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076534.309086:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923749760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076534.309097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923749248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076534.309124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076534.309128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076534.309132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923749248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076534.309165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076534.309186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076534.309216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076534/real 1769076534] req@ffff969e0a7fc000 x1855007923749376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076534.309228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923749376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076534.309237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076534.309240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076534.309243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076534.309249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076534.309267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076534.309275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076534/real 1769076534] req@ffff969e0a7fd500 x1855007923749504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076534.309286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923749504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076534.309322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076534.309325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076534.309328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076534.309332:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076534.309348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076534.309384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076534/real 1769076534] req@ffff969e0a7fdc00 x1855007923749632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076534.309394:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923749632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076534.309400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076534.309403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076534.309405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769076534.309410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076534.309424:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076534.309431:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076534/real 1769076534] req@ffff969e0a7ff800 x1855007923749760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076534.309439:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923749760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076534.309445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076534.309447:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076534.309473:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923749760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076534.628690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076535.332940:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076535.652709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076536.356777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076536.676679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076537.380754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076537.700721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076538.404916:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076538.724669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076539.428715:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076539.428732:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076539.428738:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076539.428752:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3228 00000100:00080000:3.0:1769076539.428766:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076539.428776:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076539.428788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076539.428828:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923749888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076539.428861:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076539.428866:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076539.428869:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076539.428876:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3228 00000100:00080000:3.0:1769076539.428881:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076539.428887:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076539.428891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923749888:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076539.428923:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076539.428955:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923750016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076539.428968:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076539.428974:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1769076539.428976:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076539.428978:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076539.428986:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3228 00000100:00080000:3.0:1769076539.428993:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076539.428998:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076539.429005:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076539.429030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076539/real 1769076539] req@ffff969e0a7ffb80 x1855007923749888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076539.429030:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923750144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076539.429055:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076539.429060:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076539.429061:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076539.429065:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3228 00000100:00080000:3.0:1769076539.429069:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076539.429074:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076539.429075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923749888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076539.429079:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769076539.429093:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076539.429094:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923750272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076539.429101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076539.429108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923749888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076539.429118:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076539.429121:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076539.429128:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076539.429132:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3228 00000100:00080000:3.0:1769076539.429136:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076539.429142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750016:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076539.429142:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076539.429151:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076539.429165:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923750400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076539.429174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076539.429189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076539/real 1769076539] req@ffff969e0a7ff800 x1855007923750016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076539.429198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923750016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076539.429204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076539.429206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076539.429218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076539.429223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076539.429238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076539.429245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076539/real 1769076539] req@ffff969e0a7fdc00 x1855007923750144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076539.429253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923750144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076539.429258:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076539.429261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076539.429263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076539.429284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076539.429301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076539.429308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076539/real 1769076539] req@ffff969e0a7fd500 x1855007923750272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076539.429316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923750272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076539.429322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076539.429324:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076539.429326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769076539.429339:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076539.429349:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076539.429363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076539/real 1769076539] req@ffff969e0a7fc000 x1855007923750400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076539.429371:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923750400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076539.429378:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076539.429380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076539.429382:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076539.748713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076540.452888:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076540.772711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076541.476882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076541.796707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076542.500764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076542.820648:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076543.524776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076543.844667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076544.548745:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076544.548759:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076544.548765:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076544.548778:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3233 00000100:00080000:3.0:1769076544.548793:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076544.548804:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076544.548854:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923750528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076544.548858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076544.548887:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076544.548893:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076544.548897:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076544.548904:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3233 00000100:00080000:3.0:1769076544.548908:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076544.548913:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076544.548926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923750528:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076544.548927:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076544.548950:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923750656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076544.548998:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076544.549003:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076544.549006:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076544.549011:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3233 00000100:00080000:3.0:1769076544.549018:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076544.549021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076544.549029:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076544.549048:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076544.549067:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923750784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076544.549081:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076544.549087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076544/real 1769076544] req@ffff969e0a7fca80 x1855007923750528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076544.549089:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076544.549125:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076544.549128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923750528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076544.549136:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3233 00000100:00080000:3.0:1769076544.549146:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076544.549158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076544.549158:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076544.549171:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769076544.549173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076544.549225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923750528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076544.549225:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923750912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076544.549251:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076544.549256:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076544.549258:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076544.549262:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3233 00000100:00100000:0.0:1769076544.549277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750656:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076544.549283:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076544.549295:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076544.549305:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769076544.549325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076544.549329:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923751040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076544.549376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076544/real 1769076544] req@ffff969e0a7fc380 x1855007923750656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076544.549395:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923750656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076544.549406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076544.549411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076544.549416:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076544.549424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076544.549447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076544.549459:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076544/real 1769076544] req@ffff969e0a7fe300 x1855007923750784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076544.549473:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923750784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076544.549483:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076544.549486:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076544.549490:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076544.549500:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076544.549533:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076544.549553:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076544/real 1769076544] req@ffff969e0a7fd180 x1855007923750912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076544.549578:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923750912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076544.549652:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076544.549658:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076544.549663:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923750912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769076544.549687:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076544.549717:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076544.549731:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076544/real 1769076544] req@ffff969e0a7fdf80 x1855007923751040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076544.549745:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923751040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076544.549756:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076544.549760:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076544.549764:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076544.868732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076545.572824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076545.892687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076546.596729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076546.916685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076547.620720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076547.940666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076548.644781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076548.964680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076549.668712:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076549.668721:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076549.668724:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076549.668733:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3238 00000100:00080000:0.0:1769076549.668742:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076549.668748:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076549.668773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076549.668799:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923751168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076549.668820:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076549.668822:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076549.668823:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076549.668826:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3238 00000100:00080000:0.0:1769076549.668830:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076549.668834:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076549.668841:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076549.668847:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923751168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076549.668852:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923751296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076549.668862:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076549.668866:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076549.668868:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076549.668873:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3238 00000100:00080000:0.0:1769076549.668877:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076549.668881:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076549.668887:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076549.668901:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923751424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076549.668906:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076549.668918:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076549.668923:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076549.668925:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076549.668935:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3238 00000100:00080000:0.0:1769076549.668939:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076549.668942:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076549.668947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076549/real 1769076549] req@ffff969e06a61880 x1855007923751168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076549.668949:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076549.668963:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923751552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076549.668968:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923751168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076549.668977:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076549.668983:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076549.668985:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076549.668985:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076549.668995:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076549.668996:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3238 00000100:00080000:0.0:1769076549.669002:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076549.669005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923751168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076549.669009:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076549.669014:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076549.669021:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923751680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076549.669030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076549.669048:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076549.669058:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076549/real 1769076549] req@ffff969e06a63100 x1855007923751296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076549.669078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923751296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076549.669084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076549.669087:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076549.669089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076549.669093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076549.669133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076549.669140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076549/real 1769076549] req@ffff969e06a61f80 x1855007923751424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076549.669147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923751424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076549.669153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076549.669155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076549.669157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076549.669161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076549.669172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076549.669177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076549/real 1769076549] req@ffff969e06a62680 x1855007923751552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076549.669184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923751552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076549.669200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076549.669203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076549.669205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076549.669217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076549.669229:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076549.669235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076549/real 1769076549] req@ffff969e06a61500 x1855007923751680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076549.669242:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923751680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076549.669248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076549.669250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076549.669251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076549.988700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076550.692728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076551.012720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076551.716749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076552.036674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076552.740706:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076553.060671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076553.764748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076554.084676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076554.788684:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076554.788694:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076554.788698:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076554.788706:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3243 00000100:00080000:0.0:1769076554.788716:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076554.788722:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076554.788757:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923751808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076554.788770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076554.788784:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076554.788799:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923751808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076554.788808:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076554.788812:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076554.788819:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3243 00000100:00080000:0.0:1769076554.788824:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076554.788830:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076554.788852:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076554.788869:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076554.788873:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923751936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076554.788890:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076554.788893:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076554.788896:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076554.788901:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3243 00000100:00100000:2.0:1769076554.788906:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076554/real 1769076554] req@ffff969e06a61180 x1855007923751808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076554.788908:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076554.788918:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076554.788930:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076554.788937:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923751808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076554.788942:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923752064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076554.788966:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076554.788970:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076554.788971:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076554.788972:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076554.788982:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3243 00000100:00080000:2.0:1769076554.788982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076554.788991:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076554.788993:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923751808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076554.788997:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076554.789032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751936:192.168.204.143@tcp:38:kworker.0 00000008:00100000:0.0:1769076554.789034:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076554.789051:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923752192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076554.789060:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076554.789063:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076554.789065:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769076554.789070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076554.789072:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3243 00000100:00080000:0.0:1769076554.789079:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076554.789083:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076554.789087:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076554.789095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076554/real 1769076554] req@ffff969e06a60380 x1855007923751936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076554.789097:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923752320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076554.789143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923751936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076554.789154:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076554.789158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076554.789162:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923751936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076554.789169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076554.789190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076554.789201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076554/real 1769076554] req@ffff969e06a60a80 x1855007923752064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076554.789214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923752064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076554.789223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076554.789227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076554.789230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076554.789253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076554.789274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076554.789285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076554/real 1769076554] req@ffff969e06a60700 x1855007923752192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076554.789299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923752192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076554.789308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076554.789311:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076554.789315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076554.789321:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076554.789335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076554.789344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076554/real 1769076554] req@ffff969e06a63800 x1855007923752320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076554.789356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923752320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076554.789362:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076554.789365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076554.789367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076555.108705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076555.812863:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076556.132709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076556.836762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076557.156687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076557.860783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076558.180681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076558.884764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076559.204736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076559.908729:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076559.908742:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076559.908747:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076559.908759:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3249 00000100:00080000:0.0:1769076559.908771:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076559.908781:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076559.908795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076559.908829:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923752448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076559.908854:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076559.908857:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076559.908859:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076559.908863:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3249 00000100:00080000:0.0:1769076559.908885:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076559.908894:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076559.908895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923752448:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076559.908911:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076559.908942:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923752576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076559.908965:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076559.908968:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076559.908970:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076559.908974:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3249 00000100:00080000:0.0:1769076559.908980:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076559.908983:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076559.908988:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076559.909008:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923752704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076559.909008:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076559.909025:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076559.909029:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076559.909031:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076559.909037:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3249 00000100:00080000:0.0:1769076559.909041:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076559.909052:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076559.909053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076559/real 1769076559] req@ffff969e06a60000 x1855007923752448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076559.909061:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076559.909080:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923752832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076559.909086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923752448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076559.909105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076559.909113:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076559.909114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076559.909126:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769076559.909127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923752448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076559.909134:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076559.909140:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3249 00000100:00080000:0.0:1769076559.909145:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076559.909148:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076559.909153:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076559.909157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076559.909165:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923752960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076559.909177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076559.909190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076559/real 1769076559] req@ffff969e06a62d80 x1855007923752576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076559.909199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923752576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076559.909206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076559.909209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076559.909211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076559.909216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076559.909227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076559.909232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076559/real 1769076559] req@ffff969e06a63800 x1855007923752704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076559.909254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923752704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076559.909260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076559.909262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076559.909264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076559.909268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076559.909282:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076559.909288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076559/real 1769076559] req@ffff969e06a60700 x1855007923752832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076559.909295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923752832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076559.909300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076559.909302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076559.909305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076559.909321:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076559.909339:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076559.909350:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076559/real 1769076559] req@ffff969e06a60a80 x1855007923752960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076559.909363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923752960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076559.909385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076559.909388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076559.909390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923752960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076560.228683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076560.932761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076561.252709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076561.956731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076562.276681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076562.980744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076563.300718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076564.004825:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076564.324650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076565.028713:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076565.028722:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076565.028726:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076565.028735:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3254 00000100:00080000:3.0:1769076565.028744:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769076565.028746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076565.028751:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076565.028784:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923753088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076565.028805:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076565.028808:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076565.028809:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076565.028812:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3254 00000100:00080000:3.0:1769076565.028816:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076565.028819:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076565.028827:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076565.028832:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923753088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076565.028854:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923753216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076565.028869:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076565.028887:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076565.028889:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076565.028892:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3254 00000100:00080000:3.0:1769076565.028897:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076565.028901:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076565.028905:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076565.028916:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923753344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076565.028923:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076565.028926:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076565.028928:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076565.028932:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3254 00000100:00100000:2.0:1769076565.028936:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753216:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076565.028937:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076565.028943:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076565.028948:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076565.028959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923753472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076565.028969:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076565.028971:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753344:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076565.028973:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076565.028977:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076565.028981:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3254 00000100:00080000:3.0:1769076565.028985:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076565.028988:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076565.028997:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076565.028998:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769076565.029012:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923753600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076565.029035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076565.348665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076566.052809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076566.372642:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076567.076783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076567.396671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076568.100877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1769076568.100926:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:3.0:1769076568.100971:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769076568.100990:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076568.101027:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076568.101037:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076568.101050:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076568.101061:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769076568.101073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076565/real 1769076568] req@ffff969e0a7fd880 x1855007923753472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.101101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923753472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.101113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.101134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076568.101139:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076568.101147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076565/real 1769076568] req@ffff969e0a7fea00 x1855007923753216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.101161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923753216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.101170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.101174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076568.101177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076568.101184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076565/real 1769076568] req@ffff969e0a7fc700 x1855007923753344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.101197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923753344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.101204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.101208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076568.101211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076568.101219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076565/real 1769076568] req@ffff969e0a7fce00 x1855007923753088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.101231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923753088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.101239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.101250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076568.101254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923753088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076568.101282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076565/real 1769076568] req@ffff969e0a7fe680 x1855007923753600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076568.101295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923753600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076568.101303:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076568.101306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076568.101309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076568.420656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076569.124896:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076569.444670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076570.148855:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076570.148867:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076570.148872:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076570.148884:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3259 00000100:00080000:0.0:1769076570.148897:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076570.148906:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076570.148966:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923753728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076570.148993:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076570.149017:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076570.149024:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076570.149027:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076570.149036:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3259 00000100:00080000:0.0:1769076570.149043:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076570.149051:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076570.149076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923753728:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076570.149084:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076570.149116:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923753856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076570.149139:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076570.149144:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076570.149147:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076570.149154:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3259 00000100:00080000:0.0:1769076570.149181:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076570.149192:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076570.149197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769076570.149204:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076570.149229:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923753984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076570.149247:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076570.149252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076570/real 1769076570] req@ffff969e06a60380 x1855007923753728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076570.149254:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076570.149261:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076570.149276:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3259 00000100:00080000:0.0:1769076570.149287:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076570.149296:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076570.149311:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076570.149337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923753728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076570.149349:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923754112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076570.149374:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076570.149377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076570.149380:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076570.149391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076570.149400:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076570.149407:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923753728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076570.149416:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3259 00000100:00080000:0.0:1769076570.149429:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076570.149437:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076570.149446:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076570.149459:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076570.149472:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923754240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076570.149522:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076570.149548:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076570/real 1769076570] req@ffff969e06a61180 x1855007923753856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076570.149570:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923753856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076570.149585:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076570.149627:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076570.149635:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076570.149648:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076570.149690:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076570.149712:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076570/real 1769076570] req@ffff969e06a61500 x1855007923753984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076570.149732:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923753984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076570.149798:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076570.149805:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076570.149811:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923753984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076570.149822:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076570.149865:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076570.149883:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076570/real 1769076570] req@ffff969e06a62680 x1855007923754112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076570.149903:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923754112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076570.149917:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076570.149922:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076570.149935:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076570.149963:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076570.149999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076570.150020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076570/real 1769076570] req@ffff969e06a61f80 x1855007923754240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076570.150041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923754240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076570.150055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076570.150061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076570.150079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076570.468742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076571.173089:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076571.492797:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076572.196779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076572.516815:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076573.220720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076573.540725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076574.244782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076574.564783:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076575.268746:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076575.268758:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076575.268763:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076575.268773:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3264 00000100:00080000:0.0:1769076575.268784:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076575.268792:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076575.268841:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923754368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076575.268853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076575.268871:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076575.268877:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076575.268879:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076575.268883:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3264 00000100:00080000:0.0:1769076575.268887:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076575.268890:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076575.268901:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076575.268914:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923754496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076575.268934:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923754368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076575.268964:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076575.268970:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076575.268973:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076575.268981:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3264 00000100:00080000:0.0:1769076575.268999:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076575.269007:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076575.269015:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769076575.269021:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076575.269049:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923754624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076575.269061:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076575.269068:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769076575.269069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076575/real 1769076575] req@ffff969e06a63100 x1855007923754368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076575.269071:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076575.269085:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3264 00000100:00080000:0.0:1769076575.269094:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076575.269100:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076575.269108:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076575.269111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923754368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076575.269124:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923754752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076575.269128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076575.269141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076575.269146:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076575.269152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923754368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076575.269154:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076575.269159:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076575.269169:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3264 00000100:00080000:0.0:1769076575.269181:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076575.269186:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076575.269192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754496:192.168.204.143@tcp:38:kworker.0 00000008:00100000:0.0:1769076575.269195:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076575.269215:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923754880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076575.269231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076575.269250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076575/real 1769076575] req@ffff969e06a61880 x1855007923754496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076575.269263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923754496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076575.269271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076575.269275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076575.269288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076575.269295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076575.269316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076575.269327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076575/real 1769076575] req@ffff969e06a61f80 x1855007923754624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076575.269338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923754624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076575.269346:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076575.269349:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076575.269352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076575.269357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076575.269372:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076575.269380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076575/real 1769076575] req@ffff969e06a62680 x1855007923754752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076575.269390:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923754752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076575.269398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076575.269401:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076575.269404:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076575.269419:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076575.269434:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076575.269454:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076575/real 1769076575] req@ffff969e06a61500 x1855007923754880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076575.269465:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923754880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076575.269473:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076575.269476:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076575.269479:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923754880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076575.588771:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076576.293017:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076576.612794:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076577.316810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076577.636768:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076578.340784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076578.660696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076579.364791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076579.684723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076580.388755:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076580.388773:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076580.388781:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076580.388803:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3269 00010000:00010000:2.0:1769076580.388815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076580.388819:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076580.388831:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076580.388885:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923755008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076580.388929:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076580.388934:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076580.388937:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076580.388945:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3269 00000100:00080000:0.0:1769076580.388952:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076580.388957:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076580.388971:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076580.388982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923755008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076580.388993:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923755136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076580.389041:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076580.389045:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076580.389047:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076580.389052:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3269 00000100:00080000:0.0:1769076580.389059:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076580.389063:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076580.389069:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076580.389080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076580.389090:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923755264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076580.389104:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076580.389108:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076580.389110:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076580.389115:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3269 00000100:00080000:0.0:1769076580.389118:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076580.389123:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076580.389129:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076580.389144:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923755392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076580.389152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076580/real 1769076580] req@ffff969e06a61180 x1855007923755008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076580.389157:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076580.389172:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076580.389177:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076580.389194:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3269 00000100:00080000:0.0:1769076580.389212:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076580.389221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923755008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076580.389227:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076580.389244:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076580.389254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076580.389269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076580.389275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923755008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076580.389282:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923755520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076580.389313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076580.389347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076580.389405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076580/real 1769076580] req@ffff969e06a60380 x1855007923755136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076580.389423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923755136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076580.389433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076580.389436:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076580.389440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076580.389448:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076580.389488:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076580.389500:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076580/real 1769076580] req@ffff969e06a60a80 x1855007923755264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076580.389514:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923755264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076580.389523:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076580.389527:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076580.389530:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076580.389537:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076580.389555:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076580.389565:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076580/real 1769076580] req@ffff969e06a60700 x1855007923755392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076580.389654:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923755392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076580.389671:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076580.389675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076580.389679:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076580.389685:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076580.389707:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076580.389718:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076580/real 1769076580] req@ffff969e06a63800 x1855007923755520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076580.389729:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923755520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076580.389737:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076580.389740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076580.389742:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076580.708674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076581.412777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076581.732654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076582.436751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076582.756724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076583.460816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076583.780678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076584.484729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076584.804714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076585.508729:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076585.508740:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076585.508745:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076585.508755:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3274 00000100:00080000:0.0:1769076585.508766:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076585.508773:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076585.508778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076585.508813:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923755648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076585.508840:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076585.508843:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076585.508845:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076585.508849:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3274 00000100:00080000:0.0:1769076585.508852:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076585.508855:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076585.508864:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076585.508876:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923755776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076585.508886:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076585.508895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923755648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076585.508909:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076585.508911:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076585.508925:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3274 00000100:00080000:0.0:1769076585.508929:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076585.508933:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076585.508938:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076585.508951:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923755904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076585.508961:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076585.508964:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076585.508965:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076585.508968:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3274 00000100:00080000:0.0:1769076585.508972:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076585.508975:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076585.508990:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076585.509001:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923756032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076585.509008:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076585.509015:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076585.509019:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076585.509021:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076585.509025:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3274 00000100:00080000:0.0:1769076585.509028:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076585.509032:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076585.509043:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076585.509053:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923756160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076585.509075:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076585/real 1769076585] req@ffff969e06a62d80 x1855007923755648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076585.509103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923755648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076585.509129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076585.509135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076585.509141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923755648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076585.509169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076585.509203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076585.509222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076585/real 1769076585] req@ffff969e06a60000 x1855007923755776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076585.509239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923755776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076585.509251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076585.509256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076585.509261:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076585.509270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076585.509295:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076585.509309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076585/real 1769076585] req@ffff969e06a63800 x1855007923755904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076585.509325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923755904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076585.509335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076585.509339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076585.509343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923755904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076585.509359:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076585.509386:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076585.509403:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076585/real 1769076585] req@ffff969e06a60700 x1855007923756032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076585.509429:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923756032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076585.509441:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076585.509445:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076585.509449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076585.509457:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076585.509487:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076585.509525:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076585/real 1769076585] req@ffff969e06a60a80 x1855007923756160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076585.509551:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923756160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076585.509562:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076585.509567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076585.509571:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076585.828725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076586.532865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076586.852750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076587.556833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076587.876695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076588.580725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076588.900663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076589.604835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076589.924679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076590.628816:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076590.628840:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076590.628849:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769076590.628856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076590.628873:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3279 00000100:00080000:3.0:1769076590.628914:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076590.628928:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076590.628988:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923756288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076590.629041:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076590.629046:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076590.629049:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076590.629055:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3279 00000100:00080000:3.0:1769076590.629060:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076590.629066:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076590.629097:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076590.629110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923756288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076590.629126:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923756416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076590.629145:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076590.629150:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076590.629153:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076590.629159:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3279 00000100:00080000:3.0:1769076590.629166:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076590.629171:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076590.629178:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076590.629193:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923756544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076590.629207:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076590.629212:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076590.629214:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076590.629218:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3279 00000100:00080000:3.0:1769076590.629224:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076590.629230:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076590.629235:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076590.629269:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076590.629288:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923756672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076590.629302:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076590.629307:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076590.629317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076590/real 1769076590] req@ffff969e0a7fed80 x1855007923756288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076590.629320:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076590.629335:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3279 00000100:00080000:3.0:1769076590.629346:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076590.629358:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076590.629367:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076590.629382:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923756288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076590.629389:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923756800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076590.629458:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076590.629472:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076590.629481:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923756288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076590.629523:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076590.629558:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076590.629576:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076590/real 1769076590] req@ffff969e0a7fe680 x1855007923756416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076590.629625:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923756416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076590.629637:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076590.629642:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076590.629646:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076590.629655:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076590.629691:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076590.629713:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076590/real 1769076590] req@ffff969e0a7fce00 x1855007923756544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076590.629760:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923756544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076590.629778:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076590.629785:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076590.629791:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076590.629804:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076590.629852:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076590.629875:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076590/real 1769076590] req@ffff969e0a7fc700 x1855007923756672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076590.629900:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923756672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076590.629915:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076590.629919:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076590.629923:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076590.629931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076590.629960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076590.629973:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076590/real 1769076590] req@ffff969e0a7fea00 x1855007923756800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076590.629988:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923756800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076590.630017:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076590.630022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076590.630027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923756800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076590.948742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076591.652801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076591.972697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076592.677157:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076592.996697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076593.700894:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076594.020694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076594.724817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076595.044664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076595.748952:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076595.748974:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076595.748986:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076595.748992:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076595.749034:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3284 00000100:00080000:3.0:1769076595.749062:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076595.749071:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076595.749117:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923756928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076595.749145:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076595.749149:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076595.749152:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076595.749156:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3284 00000100:00080000:3.0:1769076595.749160:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076595.749165:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076595.749175:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076595.749192:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923757056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076595.749194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923756928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076595.749232:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076595.749235:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076595.749236:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076595.749240:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3284 00000100:00080000:3.0:1769076595.749245:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076595.749248:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076595.749252:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076595.749264:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923757184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076595.749273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076595.749275:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076595.749279:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076595.749281:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076595.749285:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3284 00000100:00080000:3.0:1769076595.749287:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076595.749290:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076595.749294:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076595.749303:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923757312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076595.749310:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076595.749313:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076595.749315:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076595.749316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076595/real 1769076595] req@ffff969e0a7fd880 x1855007923756928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076595.749320:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3284 00000100:00080000:3.0:1769076595.749344:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076595.749349:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076595.749354:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076595.749361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923756928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076595.749365:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923757440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076595.749389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076595.749398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076595.749402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923756928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076595.749428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076595.749465:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076595.749480:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076595/real 1769076595] req@ffff969e0a7fdf80 x1855007923757056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076595.749494:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923757056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076595.749512:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076595.749516:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076595.749519:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076595.749525:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076595.749549:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076595.749561:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076595/real 1769076595] req@ffff969e0a7fd180 x1855007923757184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076595.749573:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923757184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076595.749581:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076595.749622:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076595.749626:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076595.749633:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076595.749658:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076595.749669:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076595/real 1769076595] req@ffff969e0a7fe300 x1855007923757312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076595.749680:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923757312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076595.749687:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076595.749690:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076595.749701:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076595.749708:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076595.749729:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076595.749742:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076595/real 1769076595] req@ffff969e0a7fc380 x1855007923757440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076595.749753:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923757440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076595.749762:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076595.749765:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076595.749768:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076596.068652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076596.772947:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076597.092650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076597.796864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076598.116685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076598.820801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076599.140666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076599.844795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076600.164680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076600.868714:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076600.868726:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076600.868732:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769076600.868750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076600.868755:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3290 00000100:00080000:0.0:1769076600.868768:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076600.868875:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076600.868948:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923757568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076600.868982:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076600.868986:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076600.868988:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076600.868993:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3290 00000100:00080000:0.0:1769076600.869003:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076600.869007:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076600.869034:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076600.869036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923757568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076600.869053:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923757696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076600.869064:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076600.869067:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076600.869069:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076600.869073:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3290 00000100:00080000:0.0:1769076600.869098:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076600.869103:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076600.869108:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076600.869127:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923757824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076600.869127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076600.869144:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076600.869149:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076600.869152:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076600.869158:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3290 00000100:00080000:0.0:1769076600.869164:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076600.869167:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076600.869174:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076600.869185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076600/real 1769076600] req@ffff969e06a60380 x1855007923757568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076600.869187:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923757952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076600.869226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923757568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076600.869231:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076600.869253:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076600.869256:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076600.869261:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3290 00000100:00080000:0.0:1769076600.869272:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076600.869277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076600.869279:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076600.869288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:0.0:1769076600.869290:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076600.869303:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923758080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076600.869312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923757568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076600.869348:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076600.869387:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076600.869397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076600/real 1769076600] req@ffff969e06a61180 x1855007923757696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076600.869407:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923757696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076600.869414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076600.869418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076600.869421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076600.869426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076600.869438:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076600.869444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076600/real 1769076600] req@ffff969e06a61500 x1855007923757824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076600.869451:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923757824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076600.869457:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076600.869459:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076600.869461:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076600.869464:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076600.869475:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076600.869494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076600/real 1769076600] req@ffff969e06a62680 x1855007923757952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076600.869507:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923757952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076600.869516:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076600.869519:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076600.869523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923757952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076600.869541:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076600.869566:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076600.869578:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076600/real 1769076600] req@ffff969e06a61f80 x1855007923758080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076600.869615:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923758080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076600.869622:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076600.869625:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076600.869627:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076601.188668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076601.892981:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076602.212718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076602.916791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076603.236693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076603.940820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076604.260682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076604.964866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076605.284684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076605.988732:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076605.988750:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076605.988775:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769076605.988780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076605.988792:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3295 00000100:00080000:0.0:1769076605.988811:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076605.988820:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076605.988898:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923758208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076605.988943:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076605.988948:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076605.988951:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076605.988956:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3295 00000100:00080000:0.0:1769076605.988961:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076605.988965:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076605.988978:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076605.988994:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923758336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076605.989026:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076605.989031:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076605.989033:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076605.989039:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3295 00000100:00080000:0.0:1769076605.989043:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076605.989050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923758208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076605.989053:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076605.989091:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076605.989113:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923758464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076605.989128:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076605.989133:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076605.989136:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076605.989141:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3295 00000100:00080000:0.0:1769076605.989177:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076605.989184:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076605.989192:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076605.989221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076605.989230:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923758592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076605.989249:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076605.989255:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076605.989257:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076605.989263:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3295 00000100:00080000:0.0:1769076605.989270:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076605.989276:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076605.989282:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076605.989297:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923758720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076605.989309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076605/real 1769076605] req@ffff969e06a61880 x1855007923758208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076605.989338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923758208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076605.989353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076605.989359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076605.989364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923758208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076605.989374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076605.989405:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076605.989420:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076605/real 1769076605] req@ffff969e06a63100 x1855007923758336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076605.989434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923758336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076605.989445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076605.989448:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076605.989452:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076605.989484:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076605.989502:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076605.989513:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076605/real 1769076605] req@ffff969e06a61f80 x1855007923758464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076605.989527:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923758464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076605.989554:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076605.989558:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076605.989562:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076605.989569:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076605.989594:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076605.989638:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076605/real 1769076605] req@ffff969e06a62680 x1855007923758592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076605.989654:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923758592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076605.989664:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076605.989668:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076605.989672:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076605.989679:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076605.989701:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076605.989713:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076605/real 1769076605] req@ffff969e06a61500 x1855007923758720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076605.989726:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923758720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076605.989736:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076605.989740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076605.989793:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076606.308760:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076607.013091:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076607.332781:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076608.036898:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076608.356694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076609.060892:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076609.380782:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076610.084969:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076610.404741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076611.108767:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076611.108776:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076611.108780:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076611.108789:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3300 00000100:00080000:3.0:1769076611.108799:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076611.108807:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076611.108819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076611.108842:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923758848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076611.108869:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076611.108873:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076611.108876:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076611.108880:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3300 00000100:00080000:3.0:1769076611.108885:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076611.108890:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076611.108910:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076611.108948:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923758848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076611.108948:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923758976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076611.109001:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076611.109022:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076611.109025:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076611.109031:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3300 00000100:00080000:3.0:1769076611.109036:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076611.109040:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076611.109046:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076611.109057:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923759104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076611.109090:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076611.109095:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076611.109097:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076611.109117:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3300 00000100:00080000:3.0:1769076611.109130:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076611.109135:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076611.109143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769076611.109144:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076611.109168:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923759232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076611.109178:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076611.109182:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076611.109184:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076611.109192:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3300 00000100:00080000:3.0:1769076611.109196:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076611.109202:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076611.109206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076611/real 1769076611] req@ffff969e0a7fca80 x1855007923758848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076611.109223:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076611.109239:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923759360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076611.109243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923758848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076611.109305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076611.109310:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076611.109315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923758848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076611.109347:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076611.109389:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076611.109406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076611/real 1769076611] req@ffff969e0a7fc000 x1855007923758976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076611.109423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923758976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076611.109454:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076611.109460:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076611.109488:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923758976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076611.109517:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076611.109559:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076611.109579:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076611/real 1769076611] req@ffff969e0a7fd500 x1855007923759104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076611.109611:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923759104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076611.109619:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076611.109622:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076611.109625:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076611.109631:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076611.109685:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076611.109704:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076611/real 1769076611] req@ffff969e0a7fdc00 x1855007923759232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076611.109723:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923759232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076611.109733:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076611.109737:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076611.109740:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076611.109746:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076611.109766:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076611.109786:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076611/real 1769076611] req@ffff969e0a7ff800 x1855007923759360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076611.109796:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923759360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076611.109805:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076611.109807:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076611.109810:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076611.428768:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076612.132856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076612.452712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076613.156856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076613.476721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076614.180841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076614.500726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076615.204867:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076615.524759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076616.228779:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076616.228792:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076616.228799:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076616.228812:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3305 00000100:00080000:3.0:1769076616.228827:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076616.228836:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076616.228872:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076616.228899:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923759488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076616.228938:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076616.228943:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076616.228946:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076616.228951:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3305 00000100:00080000:3.0:1769076616.228955:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076616.228960:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076616.228971:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076616.229020:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923759616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076616.229034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923759488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076616.229042:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076616.229049:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076616.229052:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076616.229057:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3305 00000100:00080000:3.0:1769076616.229086:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076616.229093:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076616.229101:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076616.229118:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923759744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076616.229131:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076616.229136:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076616.229140:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076616.229148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076616.229153:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3305 00000100:00080000:3.0:1769076616.229161:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076616.229166:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076616.229173:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076616.229189:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923759872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076616.229202:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076616.229207:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076616.229209:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076616.229218:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3305 00000100:00100000:1.0:1769076616.229220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076616/real 1769076616] req@ffff969e0a7ffb80 x1855007923759488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076616.229236:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076616.229245:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076616.229253:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076616.229256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923759488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076616.229281:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923760000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076616.229286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076616.229303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076616.229309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923759488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076616.229359:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076616.229394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076616.229426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076616/real 1769076616] req@ffff969e0a7ff800 x1855007923759616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076616.229446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923759616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076616.229457:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076616.229461:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076616.229466:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076616.229473:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076616.229496:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076616.229508:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076616/real 1769076616] req@ffff969e0a7fdc00 x1855007923759744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076616.229522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923759744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076616.229531:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076616.229535:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076616.229539:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076616.229545:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076616.229564:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076616.229574:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076616/real 1769076616] req@ffff969e0a7fd500 x1855007923759872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076616.229620:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923759872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076616.229642:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076616.229646:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076616.229651:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923759872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076616.229659:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076616.229687:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076616.229701:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076616/real 1769076616] req@ffff969e0a7fc000 x1855007923760000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076616.229715:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923760000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076616.229734:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076616.229738:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076616.229742:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076616.548741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076617.252858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076617.572723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076618.276773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076618.596676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076619.300800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076619.620717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076620.324804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076620.644751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076621.348806:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076621.348824:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076621.348857:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076621.348874:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3310 00010000:00010000:2.0:1769076621.348889:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076621.348917:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076621.348932:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076621.349021:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923760128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076621.349072:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076621.349080:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076621.349084:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076621.349102:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3310 00000100:00080000:0.0:1769076621.349109:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076621.349117:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076621.349134:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076621.349158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923760128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076621.349163:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923760256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076621.349201:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076621.349208:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076621.349212:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076621.349220:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3310 00000100:00080000:0.0:1769076621.349253:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076621.349262:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076621.349272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769076621.349276:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076621.349307:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923760384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076621.349323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076621/real 1769076621] req@ffff969e06a61180 x1855007923760128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076621.349351:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923760128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076621.349393:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076621.349415:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076621.349424:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076621.349428:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076621.349438:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3310 00000100:00080000:1.0:1769076621.349440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076621.349456:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923760128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076621.349458:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076621.349485:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076621.349498:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076621.349508:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076621.349532:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923760512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076621.349555:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076621.349578:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076621/real 1769076621] req@ffff969e06a60380 x1855007923760256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076621.349579:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076621.349603:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076621.349613:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076621.349656:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923760256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076621.349669:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076621.349676:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076621.349680:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076621.349690:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760384:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769076621.349711:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3310 00000100:00080000:0.0:1769076621.349723:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076621.349743:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076621.349753:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076621.349767:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076621.349768:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076621/real 1769076621] req@ffff969e06a60a80 x1855007923760384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076621.349792:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923760384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076621.349804:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923760640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076621.349807:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076621.349821:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076621.349834:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076621.349872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076621.349930:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076621.349949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076621/real 1769076621] req@ffff969e06a60700 x1855007923760512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076621.349970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923760512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076621.349981:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076621.349985:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076621.349988:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076621.349995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076621.350015:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076621.350026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076621/real 1769076621] req@ffff969e06a63800 x1855007923760640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076621.350038:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923760640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076621.350065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076621.350068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076621.350072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076621.668731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076622.372820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076622.692679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076623.396817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076623.716792:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076624.420801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076624.740702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076625.444912:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076625.764725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076626.468796:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076626.468863:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076626.468871:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076626.468892:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3315 00000100:00080000:0.0:1769076626.468966:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076626.468981:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076626.469015:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076626.469045:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923760768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076626.469109:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076626.469113:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076626.469116:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076626.469121:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3315 00000100:00080000:0.0:1769076626.469126:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076626.469130:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076626.469144:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076626.469183:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923760896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076626.469203:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076626.469214:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076626.469214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923760768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076626.469242:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076626.469249:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3315 00000100:00080000:0.0:1769076626.469253:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076626.469258:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076626.469266:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076626.469284:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923761024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076626.469295:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076626.469309:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076626.469312:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076626.469318:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3315 00000100:00080000:0.0:1769076626.469325:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076626.469329:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076626.469341:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769076626.469347:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076626.469371:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923761152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076626.469386:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076626.469391:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076626.469405:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076626/real 1769076626] req@ffff969e06a60000 x1855007923760768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076626.469406:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076626.469431:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3315 00000100:00080000:0.0:1769076626.469448:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076626.469461:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076626.469472:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076626.469475:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923760768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076626.469490:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923761280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076626.469498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076626.469515:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076626.469523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923760768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076626.469568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076626.469656:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076626.469687:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076626/real 1769076626] req@ffff969e06a62d80 x1855007923760896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076626.469720:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923760896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076626.469732:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076626.469740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076626.469750:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923760896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076626.469762:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076626.469784:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076626.469796:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076626/real 1769076626] req@ffff969e06a63800 x1855007923761024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076626.469874:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923761024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076626.469884:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076626.469889:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076626.469893:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076626.469902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076626.469952:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076626.469975:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076626/real 1769076626] req@ffff969e06a60700 x1855007923761152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076626.470000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923761152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076626.470018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076626.470025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076626.470032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076626.470044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076626.470078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076626.470093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076626/real 1769076626] req@ffff969e06a60a80 x1855007923761280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076626.470107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923761280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076626.470136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076626.470141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076626.470146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076626.788742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076627.492898:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076627.812699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076628.516927:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076628.836732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076629.540886:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076629.860714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076630.564819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076630.884836:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076631.588765:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076631.588777:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076631.588782:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076631.588792:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3320 00000100:00080000:3.0:1769076631.588804:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076631.588811:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076631.588820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076631.588852:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923761408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076631.588882:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076631.588886:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076631.588888:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076631.588892:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3320 00000100:00080000:3.0:1769076631.588895:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076631.588899:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076631.588909:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076631.588921:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923761536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076631.588929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923761408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076631.588936:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076631.588940:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076631.588942:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076631.588945:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3320 00000100:00080000:3.0:1769076631.588948:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076631.588951:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076631.588974:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076631.588986:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923761664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076631.588996:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076631.588999:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076631.589001:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076631.589004:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3320 00000100:00080000:3.0:1769076631.589008:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076631.589012:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076631.589017:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076631.589030:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923761792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076631.589045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761536:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076631.589049:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076631.589059:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076631.589062:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076631.589081:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3320 00000100:00080000:3.0:1769076631.589088:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076631.589095:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076631.589104:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076631.589115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076631.589126:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923761920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076631.589177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076631.589226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076631.908704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076632.612812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076632.932723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076633.636863:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076633.956671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076634.660824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769076634.660938:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769076634.661004:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076634.661023:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076634.661078:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076634.661109:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076634.661122:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076634.661138:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076634.661198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076631/real 1769076634] req@ffff969e0a7fd180 x1855007923761792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076634.661237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923761792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076634.661254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076634.661261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076634.661280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076634.661292:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076631/real 1769076634] req@ffff969e0a7fdf80 x1855007923761920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076634.661326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923761920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076634.661339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076634.661343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076634.661349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076634.661358:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076631/real 1769076634] req@ffff969e0a7fc380 x1855007923761536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076634.661376:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923761536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076634.661388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076634.661393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076634.661398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076634.661408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076631/real 1769076634] req@ffff969e0a7fe300 x1855007923761664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076634.661425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923761664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076634.661437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076634.661443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076634.661458:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923761664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076634.661478:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076631/real 1769076634] req@ffff969e0a7fca80 x1855007923761408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076634.661497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923761408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076634.661508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076634.661514:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076634.661519:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923761408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076634.980739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076635.684833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076636.004654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076636.708706:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076636.708716:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076636.708751:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076636.708761:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3325 00010000:00010000:0.0:1769076636.708767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076636.708774:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076636.708783:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076636.708834:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923762048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076636.708859:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076636.708861:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076636.708863:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076636.708866:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3325 00000100:00080000:3.0:1769076636.708868:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076636.708870:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076636.708878:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076636.708888:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923762176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076636.708895:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076636.708898:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076636.708899:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076636.708901:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3325 00000100:00080000:3.0:1769076636.708903:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076636.708908:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923762048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076636.708909:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076636.708918:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076636.708926:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923762304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076636.708943:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076636.708946:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076636.708947:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076636.708950:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3325 00000100:00080000:3.0:1769076636.708954:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076636.708956:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076636.708961:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076636.708972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076636.708973:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923762432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076636.708988:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076636.708991:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076636.708993:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076636.708996:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3325 00000100:00080000:3.0:1769076636.709000:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076636.709003:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076636.709006:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076636.709012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076636/real 1769076636] req@ffff969e0a7fd880 x1855007923762048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076636.709018:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923762560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076636.709051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923762048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076636.709063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076636.709068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076636.709072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923762048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076636.709080:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076636.709105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076636.709117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076636/real 1769076636] req@ffff969e0a7fea00 x1855007923762176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076636.709140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923762176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076636.709148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076636.709151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076636.709154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076636.709179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076636.709201:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076636.709213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076636/real 1769076636] req@ffff969e0a7fc700 x1855007923762304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076636.709226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923762304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076636.709234:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076636.709237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076636.709241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076636.709248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076636.709268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076636.709280:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076636/real 1769076636] req@ffff969e0a7fce00 x1855007923762432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076636.709292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923762432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076636.709317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076636.709320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076636.709323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076636.709330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076636.709350:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076636.709361:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076636/real 1769076636] req@ffff969e0a7fe680 x1855007923762560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076636.709372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923762560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076636.709380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076636.709383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076636.709386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076637.028833:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076637.732811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076638.052738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076638.756751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076639.076720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076639.780841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076640.100647:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076640.804815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076641.124727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076641.828742:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076641.828756:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076641.828762:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076641.828774:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3331 00000100:00080000:0.0:1769076641.828787:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076641.828796:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076641.828843:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923762688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076641.828884:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076641.828885:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076641.828895:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076641.828899:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076641.828906:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3331 00000100:00080000:0.0:1769076641.828912:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076641.828917:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076641.828928:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076641.828968:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923762816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076641.828982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923762688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076641.828990:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076641.828997:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076641.829016:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076641.829024:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3331 00000100:00080000:0.0:1769076641.829031:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076641.829037:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076641.829044:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076641.829062:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923762944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076641.829078:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076641.829085:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076641.829091:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076641.829094:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076641.829101:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3331 00000100:00080000:0.0:1769076641.829108:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076641.829113:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076641.829120:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076641.829132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076641/real 1769076641] req@ffff969e06a60380 x1855007923762688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076641.829139:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923763072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076641.829189:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076641.829194:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076641.829199:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076641.829201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923762688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076641.829207:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3331 00000100:00080000:0.0:1769076641.829211:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076641.829218:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076641.829236:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076641.829243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076641.829252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076641.829263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923762688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076641.829288:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923763200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076641.829301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076641.829351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076641.829371:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076641/real 1769076641] req@ffff969e06a61180 x1855007923762816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076641.829387:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923762816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076641.829397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076641.829402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076641.829406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076641.829413:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076641.829432:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076641.829442:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076641/real 1769076641] req@ffff969e06a61500 x1855007923762944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076641.829456:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923762944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076641.829466:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076641.829469:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076641.829473:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923762944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076641.829491:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076641.829533:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076641.829567:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076641/real 1769076641] req@ffff969e06a62680 x1855007923763072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076641.829584:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923763072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076641.829625:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076641.829630:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076641.829634:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076641.829657:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076641.829682:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076641.829695:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076641/real 1769076641] req@ffff969e06a61f80 x1855007923763200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076641.829709:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923763200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076641.829719:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076641.829723:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076641.829727:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076642.148731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076642.852819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076643.172676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076643.876771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076644.196687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076644.900799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076645.220700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076645.924720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076646.244673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076646.948688:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076646.948697:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076646.948701:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076646.948710:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3336 00000100:00080000:0.0:1769076646.948719:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076646.948726:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076646.948756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076646.948759:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923763328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076646.948781:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076646.948784:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076646.948785:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076646.948788:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3336 00000100:00080000:0.0:1769076646.948791:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076646.948794:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076646.948802:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076646.948814:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923763456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076646.948814:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923763328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076646.948828:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076646.948830:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076646.948831:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076646.948834:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3336 00000100:00080000:0.0:1769076646.948836:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076646.948853:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076646.948857:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076646.948866:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923763584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076646.948873:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076646.948876:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076646.948879:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076646.948879:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076646.948884:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3336 00000100:00080000:0.0:1769076646.948889:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076646.948892:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076646.948897:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076646.948905:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923763712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076646.948912:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076646.948913:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076646/real 1769076646] req@ffff969e06a63100 x1855007923763328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076646.948915:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076646.948918:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076646.948925:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3336 00000100:00080000:0.0:1769076646.948930:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076646.948934:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076646.948937:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076646.948941:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923763328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076646.948946:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923763840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076646.948959:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076646.948966:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076646.948969:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923763328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076646.948988:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076646.949002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076646.949009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076646/real 1769076646] req@ffff969e06a61880 x1855007923763456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076646.949017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923763456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076646.949022:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076646.949025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076646.949027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076646.949030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076646.949040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076646.949045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076646/real 1769076646] req@ffff969e06a61f80 x1855007923763584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076646.949060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923763584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076646.949065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076646.949067:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076646.949069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076646.949073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076646.949085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076646.949091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076646/real 1769076646] req@ffff969e06a62680 x1855007923763712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076646.949097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923763712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076646.949103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076646.949104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076646.949106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076646.949109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076646.949118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076646.949123:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076646/real 1769076646] req@ffff969e06a61500 x1855007923763840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076646.949130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923763840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076646.949144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076646.949146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076646.949149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923763840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076647.268651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076647.972731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076648.292666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076648.996781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076649.316721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076650.020814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076650.340706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076651.044732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076651.364685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076652.068690:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076652.068716:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076652.068720:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076652.068728:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3341 00000100:00080000:0.0:1769076652.068738:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076652.068744:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076652.068758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076652.068775:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923763968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076652.068797:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076652.068799:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076652.068801:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076652.068804:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3341 00000100:00080000:0.0:1769076652.068806:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076652.068809:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076652.068816:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076652.068826:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923764096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076652.068830:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923763968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076652.068837:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076652.068840:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076652.068841:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076652.068844:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3341 00000100:00080000:0.0:1769076652.068846:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076652.068848:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076652.068851:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076652.068859:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923764224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076652.068871:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076652.068873:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076652.068875:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076652.068877:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3341 00000100:00080000:0.0:1769076652.068881:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076652.068883:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076652.068887:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076652.068892:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076652.068899:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923764352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076652.068907:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076652.068909:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076652.068910:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076652.068913:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3341 00000100:00080000:0.0:1769076652.068916:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076652.068918:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076652.068922:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076652.068923:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076652/real 1769076652] req@ffff969e06a61180 x1855007923763968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076652.068931:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923764480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076652.068950:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923763968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076652.068959:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076652.068963:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076652.068966:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923763968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076652.068984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076652.068999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076652.069007:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076652/real 1769076652] req@ffff969e06a60380 x1855007923764096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076652.069015:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923764096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076652.069027:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076652.069030:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076652.069032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076652.069036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076652.069050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076652.069056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076652/real 1769076652] req@ffff969e06a60a80 x1855007923764224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076652.069063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923764224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076652.069069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076652.069071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076652.069073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076652.069076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076652.069086:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076652.069092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076652/real 1769076652] req@ffff969e06a60700 x1855007923764352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076652.069099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923764352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076652.069104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076652.069106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076652.069112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076652.069116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076652.069129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076652.069135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076652/real 1769076652] req@ffff969e06a63800 x1855007923764480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076652.069142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923764480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076652.069148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076652.069149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076652.069151:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076652.388693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076653.092716:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076653.412700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076654.116808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076654.436665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076655.140761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076655.460720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076656.164746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076656.484669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076657.188704:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076657.188713:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076657.188716:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076657.188724:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3346 00000100:00080000:3.0:1769076657.188753:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076657.188762:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076657.188790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076657.188799:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923764608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076657.188822:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076657.188825:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076657.188826:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076657.188830:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3346 00000100:00080000:3.0:1769076657.188833:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076657.188837:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076657.188845:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076657.188856:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923764608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076657.188859:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923764736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076657.188869:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076657.188871:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076657.188872:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076657.188875:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3346 00000100:00080000:3.0:1769076657.188878:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076657.188880:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076657.188883:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076657.188892:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923764864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076657.188898:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076657.188907:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076657.188909:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076657.188911:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3346 00000100:00080000:3.0:1769076657.188915:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076657.188920:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076657.188921:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076657.188928:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076657.188938:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923764992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076657.188945:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076657.188948:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076657.188949:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076657.188954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076657/real 1769076657] req@ffff969e0a7fed80 x1855007923764608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076657.188955:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3346 00000100:00080000:3.0:1769076657.188960:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076657.188965:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076657.188974:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076657.188979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923764608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076657.188988:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923765120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076657.188990:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076657.188998:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076657.189011:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923764608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076657.189030:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076657.189047:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076657.189055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076657/real 1769076657] req@ffff969e0a7fe680 x1855007923764736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076657.189064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923764736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076657.189069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076657.189072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076657.189074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076657.189077:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076657.189087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076657.189092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076657/real 1769076657] req@ffff969e0a7fce00 x1855007923764864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076657.189099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923764864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076657.189104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076657.189106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076657.189108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076657.189111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076657.189121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076657.189133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076657/real 1769076657] req@ffff969e0a7fc700 x1855007923764992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076657.189140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923764992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076657.189146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076657.189148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076657.189149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923764992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076657.189153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076657.189164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076657.189169:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076657/real 1769076657] req@ffff969e0a7fea00 x1855007923765120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076657.189176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923765120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076657.189182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076657.189184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076657.189185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076657.508685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076658.212711:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076658.532672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076659.237226:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076659.556669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076660.261098:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076660.580667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076661.285073:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076661.604690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076662.308962:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076662.308984:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076662.308990:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076662.309001:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3351 00000100:00080000:2.0:1769076662.309012:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076662.309021:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076662.309025:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076662.309060:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923765248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076662.309103:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076662.309108:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076662.309111:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076662.309117:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3351 00000100:00080000:2.0:1769076662.309121:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076662.309126:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076662.309136:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923765248:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076662.309137:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076662.309158:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923765376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076662.309171:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076662.309175:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076662.309178:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076662.309182:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3351 00000100:00080000:2.0:1769076662.309187:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076662.309190:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076662.309195:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076662.309225:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923765504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076662.309236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076662.309246:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076662.309252:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076662.309255:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076662.309261:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3351 00000100:00080000:2.0:1769076662.309266:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076662.309271:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076662.309283:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076662.309291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076662/real 1769076662] req@ffff969e0a46ed80 x1855007923765248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076662.309300:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923765632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076662.309333:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076662.309338:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076662.309340:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076662.309346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923765248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076662.309348:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3351 00000100:00080000:2.0:1769076662.309356:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076662.309365:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076662.309366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076662.309371:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076662.309375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076662.309381:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923765248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076662.309409:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076662.309415:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923765760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076662.309436:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076662.309450:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076662/real 1769076662] req@ffff969e0a46d880 x1855007923765376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076662.309460:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923765376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076662.309467:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076662.309469:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076662.309472:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076662.309476:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076662.309487:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076662.309494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076662/real 1769076662] req@ffff969e0a46ea00 x1855007923765504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076662.309501:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923765504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076662.309517:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076662.309519:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076662.309522:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076662.309526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076662.309540:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076662.309547:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076662/real 1769076662] req@ffff969e0a46ca80 x1855007923765632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076662.309554:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923765632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076662.309559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076662.309561:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076662.309563:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076662.309575:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076662.309588:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076662.309627:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076662/real 1769076662] req@ffff969e0a46df80 x1855007923765760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076662.309636:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923765760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076662.309643:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076662.309645:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076662.309661:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923765760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076662.628669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076663.332902:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076663.652725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076664.356713:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076664.676654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076665.380777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076665.700687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076666.404765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076666.724658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076667.428699:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076667.428711:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076667.428732:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076667.428743:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3356 00010000:00010000:2.0:1769076667.428751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076667.428754:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076667.428763:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076667.428798:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923765888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076667.428828:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076667.428832:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076667.428834:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076667.428839:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3356 00000100:00080000:3.0:1769076667.428844:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076667.428849:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076667.428858:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076667.428865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923765888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076667.428875:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923766016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076667.428887:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076667.428891:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076667.428893:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076667.428897:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3356 00000100:00080000:3.0:1769076667.428902:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076667.428906:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076667.428911:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076667.428924:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923766144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076667.428930:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076667.428954:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076667.428958:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076667.428960:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076667.428966:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3356 00000100:00100000:1.0:1769076667.428970:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076667/real 1769076667] req@ffff969e0a7fd880 x1855007923765888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076667.428973:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076667.428982:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076667.428991:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076667.429000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923765888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076667.429008:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923766272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076667.429018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076667.429022:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076667.429027:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076667.429029:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076667.429034:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076667.429038:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923765888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076667.429042:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3356 00000100:00080000:3.0:1769076667.429048:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076667.429053:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076667.429057:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076667.429069:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923766400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076667.429071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076667.429104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076667.429131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076667/real 1769076667] req@ffff969e0a7fca80 x1855007923766016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076667.429146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923766016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076667.429156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076667.429160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076667.429172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076667.429179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076667.429200:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076667.429210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076667/real 1769076667] req@ffff969e0a7fe300 x1855007923766144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076667.429223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923766144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076667.429231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076667.429235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076667.429238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076667.429245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076667.429261:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076667.429270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076667/real 1769076667] req@ffff969e0a7fc380 x1855007923766272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076667.429282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923766272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076667.429292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076667.429295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076667.429298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076667.429314:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076667.429331:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076667.429351:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076667/real 1769076667] req@ffff969e0a7fdf80 x1855007923766400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076667.429364:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923766400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076667.429373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076667.429376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076667.429380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076667.748701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076668.452749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076668.772709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076669.476729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076669.796682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076670.500740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076670.820685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076671.524739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076671.844647:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076672.548707:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076672.548719:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076672.548725:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076672.548738:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3361 00000100:00080000:2.0:1769076672.548752:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076672.548764:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076672.548765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076672.548819:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923766528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076672.548847:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076672.548850:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076672.548852:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076672.548855:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3361 00000100:00080000:2.0:1769076672.548860:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076672.548864:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076672.548873:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076672.548885:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923766656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076672.548894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923766528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076672.548897:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076672.548909:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076672.548912:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076672.548915:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3361 00000100:00080000:2.0:1769076672.548918:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076672.548922:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076672.548925:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076672.548956:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923766784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076672.548964:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076672.548966:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076672.548968:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076672.548971:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3361 00000100:00080000:2.0:1769076672.548974:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076672.548979:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076672.548984:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076672.548992:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923766912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076672.549001:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076672.549021:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076672.549024:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076672.549026:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076672.549047:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3361 00000100:00080000:2.0:1769076672.549052:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076672.549059:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076672.549064:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076672.549071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076672/real 1769076672] req@ffff969e0a46ce00 x1855007923766528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076672.549077:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923767040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076672.549137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923766528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076672.549150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076672.549157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076672.549161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923766528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076672.549188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076672.549207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076672.549216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076672/real 1769076672] req@ffff969e0a46e300 x1855007923766656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076672.549224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923766656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076672.549239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076672.549241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076672.549244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076672.549248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076672.549264:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076672.549271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076672/real 1769076672] req@ffff969e0a46df80 x1855007923766784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076672.549279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923766784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076672.549284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076672.549286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076672.549288:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076672.549292:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076672.549303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076672.549309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076672/real 1769076672] req@ffff969e0a46ca80 x1855007923766912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076672.549317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923766912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076672.549323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076672.549325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076672.549334:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923766912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076672.549339:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076672.549353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076672.549360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076672/real 1769076672] req@ffff969e0a46ea00 x1855007923767040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076672.549367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923767040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076672.549373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076672.549375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076672.549377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076672.868754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076673.572866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076673.892723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076674.596873:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076674.916700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076675.620839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076675.940699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076676.644752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076676.964658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076677.668720:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076677.668734:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076677.668741:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076677.668753:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3366 00000100:00080000:2.0:1769076677.668769:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076677.668778:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076677.668828:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923767168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769076677.668886:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076677.668887:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076677.668893:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076677.668896:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076677.668901:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3366 00000100:00080000:2.0:1769076677.668905:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076677.668910:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076677.668919:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076677.668933:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923767296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076677.668935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923767168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076677.668944:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076677.668961:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076677.668962:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076677.668966:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3366 00000100:00080000:2.0:1769076677.668969:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076677.668973:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076677.669005:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076677.669016:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923767424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076677.669021:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076677.669026:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076677.669029:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076677.669031:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076677.669034:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3366 00000100:00080000:2.0:1769076677.669046:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076677.669050:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076677.669053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076677/real 1769076677] req@ffff969e0a46d880 x1855007923767168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076677.669055:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076677.669064:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923767552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076677.669071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923767168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076677.669073:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076677.669076:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076677.669078:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076677.669082:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3366 00000100:00080000:1.0:1769076677.669083:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076677.669085:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076677.669087:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076677.669088:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076677.669091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923767168:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076677.669092:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076677.669100:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923767680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076677.669115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076677.669138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076677.669158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076677/real 1769076677] req@ffff969e0a46ed80 x1855007923767296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076677.669167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923767296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076677.669173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076677.669175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076677.669177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076677.669182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076677.669216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076677.669227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076677/real 1769076677] req@ffff969e0a46f480 x1855007923767424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076677.669238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923767424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076677.669246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076677.669249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076677.669252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076677.669257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076677.669273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076677.669281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076677/real 1769076677] req@ffff969e0a46d500 x1855007923767552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076677.669292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923767552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076677.669319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076677.669323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076677.669326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076677.669332:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076677.669351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076677.669361:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076677/real 1769076677] req@ffff969e0a46c380 x1855007923767680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076677.669372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923767680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076677.669380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076677.669383:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076677.669386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076677.988688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076678.692815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076679.012758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076679.716732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076680.036717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076680.740802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076681.060791:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076681.764837:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076682.084659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076682.788840:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076682.788853:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076682.788859:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769076682.788885:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076682.788917:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3371 00000100:00080000:3.0:1769076682.788929:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076682.788935:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076682.789004:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923767808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076682.789036:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076682.789041:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076682.789043:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076682.789048:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3371 00000100:00080000:3.0:1769076682.789052:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076682.789056:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076682.789067:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076682.789081:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923767936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076682.789096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923767808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076682.789124:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076682.789129:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076682.789131:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076682.789138:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3371 00000100:00080000:3.0:1769076682.789152:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076682.789157:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076682.789163:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076682.789178:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923768064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076682.789208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076682.789210:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076682.789215:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076682.789217:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076682.789221:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3371 00000100:00080000:3.0:1769076682.789226:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076682.789230:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076682.789235:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076682.789250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076682/real 1769076682] req@ffff969e0a7fd180 x1855007923767808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076682.789253:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923768192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076682.789309:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076682.789318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923767808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076682.789320:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076682.789328:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076682.789344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076682.789346:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3371 00000100:00080000:1.0:1769076682.789353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076682.789372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923767808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076682.789379:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076682.789393:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076682.789401:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076682.789425:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076682.789426:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923768320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076682.789475:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076682.789517:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076682/real 1769076682] req@ffff969e0a7fc000 x1855007923767936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076682.789528:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923767936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076682.789537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076682.789540:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076682.789543:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923767936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076682.789548:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076682.789591:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076682.789601:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076682/real 1769076682] req@ffff969e0a7fd500 x1855007923768064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076682.789613:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923768064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076682.789631:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076682.789634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076682.789637:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076682.789643:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076682.789703:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076682.789726:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076682/real 1769076682] req@ffff969e0a7fdc00 x1855007923768192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076682.789762:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923768192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076682.789783:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076682.789787:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076682.789790:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076682.789810:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076682.789833:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076682.789844:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076682/real 1769076682] req@ffff969e0a7ff800 x1855007923768320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076682.789856:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923768320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076682.789917:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076682.789923:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076682.789929:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076683.108709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076683.812824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076684.132672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076684.836786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076685.156710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076685.860971:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076686.180841:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076686.884874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076687.204862:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076687.908845:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076687.908857:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076687.908863:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076687.908876:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3377 00000100:00080000:2.0:1769076687.908899:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076687.908909:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076687.908913:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076687.908978:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923768448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076687.909015:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076687.909020:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076687.909023:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076687.909028:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3377 00000100:00080000:2.0:1769076687.909033:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076687.909039:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076687.909052:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076687.909078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923768448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076687.909080:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923768576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076687.909121:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076687.909126:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076687.909129:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076687.909137:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3377 00000100:00080000:2.0:1769076687.909144:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076687.909150:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076687.909185:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076687.909186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076687.909220:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923768704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076687.909234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076687/real 1769076687] req@ffff969e0a46f100 x1855007923768448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076687.909266:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076687.909279:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076687.909284:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076687.909289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923768448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076687.909299:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3377 00000100:00080000:2.0:1769076687.909312:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076687.909329:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076687.909345:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076687.909347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076687.909364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076687.909369:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923768832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076687.909383:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923768448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076687.909391:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076687.909402:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076687.909406:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076687.909413:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3377 00000100:00080000:2.0:1769076687.909446:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076687.909452:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076687.909459:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768576:192.168.204.143@tcp:38:kworker.0 00000008:00100000:2.0:1769076687.909464:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076687.909486:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923768960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076687.909540:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076687.909575:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076687/real 1769076687] req@ffff969e0a46f800 x1855007923768576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076687.909593:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923768576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076687.909605:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076687.909626:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076687.909630:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076687.909638:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076687.909659:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076687.909724:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076687/real 1769076687] req@ffff969e0a46c380 x1855007923768704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076687.909751:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923768704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076687.909768:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076687.909801:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076687.909810:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076687.909856:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076687.909890:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076687.909910:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076687/real 1769076687] req@ffff969e0a46d500 x1855007923768832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076687.909962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923768832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076687.909980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076687.909988:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076687.909995:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076687.910032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076687.910075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076687.910098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076687/real 1769076687] req@ffff969e0a46f480 x1855007923768960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076687.910124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923768960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076687.910142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076687.910149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076687.910153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923768960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076688.228797:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076688.932915:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076689.252704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076689.956896:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076690.276693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076690.980865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076691.300749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076692.004882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076692.324694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076693.028729:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769076693.028736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076693.028738:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076693.028743:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076693.028752:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3382 00000100:00080000:0.0:1769076693.028785:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076693.028792:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076693.028829:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923769088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076693.028855:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076693.028858:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076693.028859:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076693.028863:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3382 00000100:00080000:0.0:1769076693.028866:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076693.028869:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076693.028878:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076693.028888:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923769216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076693.028897:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076693.028898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923769088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076693.028900:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076693.028903:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076693.028910:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3382 00000100:00080000:0.0:1769076693.028912:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076693.028915:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076693.028918:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076693.028927:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923769344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076693.028933:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076693.028952:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076693.028954:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076693.028957:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3382 00000100:00080000:0.0:1769076693.028968:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076693.028971:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076693.028975:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076693.028979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076693.028986:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923769472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076693.028994:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076693.028997:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076693.028998:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076693.029001:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3382 00000100:00080000:0.0:1769076693.029003:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076693.029006:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076693.029009:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076693.029015:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076693/real 1769076693] req@ffff969e06a62d80 x1855007923769088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076693.029019:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923769600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076693.029057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923769088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076693.029066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076693.029069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076693.029072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923769088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076693.029089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076693.029104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076693.029112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076693/real 1769076693] req@ffff969e06a60000 x1855007923769216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076693.029120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923769216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076693.029136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076693.029139:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076693.029141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076693.029144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076693.029154:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076693.029160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076693/real 1769076693] req@ffff969e06a63800 x1855007923769344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076693.029175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923769344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076693.029181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076693.029183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076693.029185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076693.029189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076693.029202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076693.029208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076693/real 1769076693] req@ffff969e06a60700 x1855007923769472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076693.029215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923769472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076693.029221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076693.029222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076693.029224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076693.029228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076693.029237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076693.029242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076693/real 1769076693] req@ffff969e06a60a80 x1855007923769600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076693.029248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923769600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076693.029263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076693.029265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076693.029267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076693.348735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076694.052829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076694.372772:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076695.076806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076695.396749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076696.100797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076696.420690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076697.124817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076697.444762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076698.148732:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076698.148741:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076698.148746:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076698.148754:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3387 00000100:00080000:3.0:1769076698.148765:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076698.148772:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076698.148790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076698.148812:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923769728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076698.148839:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076698.148842:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076698.148843:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076698.148861:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3387 00000100:00080000:3.0:1769076698.148865:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076698.148868:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076698.148874:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923769728:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076698.148876:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076698.148892:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923769856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076698.148901:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076698.148904:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076698.148905:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076698.148908:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3387 00000100:00080000:3.0:1769076698.148911:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076698.148913:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076698.148917:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076698.148924:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923769984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076698.148930:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076698.148932:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076698.148934:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076698.148936:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3387 00000100:00080000:3.0:1769076698.148939:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076698.148952:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076698.148956:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076698.148965:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923770112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076698.148977:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769856:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076698.148979:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076698.148985:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076698.148987:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076698.148991:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3387 00000100:00080000:3.0:1769076698.148993:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076698.148996:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076698.149000:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076698.149012:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923770240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076698.149020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076698.149043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076698.149057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076698.468675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076699.172785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076699.492748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076700.196816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076700.516671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076701.220822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769076701.220995:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769076701.221043:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076701.221066:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076701.221118:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076701.221129:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076701.221142:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769076701.221156:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076701.221194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076698/real 1769076701] req@ffff969e0a7fd500 x1855007923770112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076701.221238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923770112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076701.221273:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076701.221283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076701.221318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076701.221333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076698/real 1769076701] req@ffff969e0a7fc000 x1855007923770240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076701.221361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923770240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076701.221379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076701.221386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076701.221394:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076701.221407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076698/real 1769076701] req@ffff969e0a7ff800 x1855007923769856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076701.221434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923769856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076701.221451:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076701.221458:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076701.221465:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076701.221477:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076698/real 1769076701] req@ffff969e0a7fdc00 x1855007923769984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076701.221503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923769984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076701.221520:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076701.221527:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076701.221548:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923769984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076701.221665:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076698/real 1769076701] req@ffff969e0a7ffb80 x1855007923769728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076701.221702:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923769728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076701.221717:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076701.221725:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076701.221732:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923769728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076701.540723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076702.244801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076702.564752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076703.268796:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076703.268810:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076703.268818:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769076703.268821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076703.268839:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3392 00000100:00080000:0.0:1769076703.268859:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076703.268883:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076703.268937:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923770368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076703.269013:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076703.269018:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076703.269021:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076703.269027:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3392 00000100:00080000:0.0:1769076703.269033:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076703.269040:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076703.269055:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076703.269083:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923770496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076703.269116:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923770368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076703.269138:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076703.269147:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076703.269151:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076703.269160:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3392 00000100:00080000:0.0:1769076703.269168:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076703.269175:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076703.269185:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076703.269210:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923770624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076703.269229:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076703.269247:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076703.269250:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076703.269259:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3392 00000800:00000100:1.0:1769076703.269266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076703.269296:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076703.269309:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076703.269321:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076703.269326:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076703/real 1769076703] req@ffff969e06a60380 x1855007923770368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076703.269352:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923770752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076703.269371:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923770368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076703.269378:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076703.269398:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076703.269406:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076703.269415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076703.269422:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3392 00000100:00080000:0.0:1769076703.269434:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076703.269443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076703.269453:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076703.269462:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923770368:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076703.269469:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076703.269499:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923770880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076703.269554:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076703.269656:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076703.269688:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076703/real 1769076703] req@ffff969e06a61180 x1855007923770496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076703.269733:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923770496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076703.269749:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076703.269756:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076703.269763:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076703.269775:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076703.269818:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076703.269839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076703/real 1769076703] req@ffff969e06a61500 x1855007923770624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076703.269861:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923770624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076703.269874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076703.269880:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076703.269887:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076703.269898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076703.269935:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076703.269957:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076703/real 1769076703] req@ffff969e06a62680 x1855007923770752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076703.269979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923770752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076703.270018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076703.270026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076703.270033:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076703.270067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076703.270130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076703.270156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076703/real 1769076703] req@ffff969e06a61f80 x1855007923770880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076703.270180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923770880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076703.270196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076703.270202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076703.270208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923770880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076703.588694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076704.292866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076704.612688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076705.316808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076705.636715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076706.340819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076706.660772:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076707.364775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076707.684667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076708.388719:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076708.388736:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076708.388744:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076708.388761:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3397 00000100:00080000:0.0:1769076708.388792:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769076708.388799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076708.388800:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076708.388862:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923771008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076708.388901:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076708.388942:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076708.388944:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076708.388948:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3397 00000100:00080000:0.0:1769076708.388952:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076708.388956:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076708.388961:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923771008:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076708.388967:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076708.388986:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923771136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076708.388999:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076708.389003:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076708.389005:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076708.389010:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3397 00000100:00080000:0.0:1769076708.389014:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076708.389018:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076708.389024:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076708.389027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076708.389042:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923771264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076708.389051:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076708.389054:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769076708.389057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076708/real 1769076708] req@ffff969e06a61880 x1855007923771008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076708.389058:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076708.389065:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3397 00000100:00080000:0.0:1769076708.389071:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076708.389076:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076708.389081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923771008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076708.389087:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076708.389100:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923771392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076708.389100:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076708.389129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076708.389142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923771008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076708.389156:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076708.389182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771136:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769076708.389190:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076708.389198:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076708.389225:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3397 00000100:00080000:0.0:1769076708.389231:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076708.389236:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076708.389242:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076708.389251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076708.389255:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923771520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076708.389270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076708/real 1769076708] req@ffff969e06a63100 x1855007923771136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076708.389287:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923771136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076708.389296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076708.389299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076708.389301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076708.389306:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076708.389329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076708.389337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076708/real 1769076708] req@ffff969e06a61f80 x1855007923771264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076708.389345:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923771264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076708.389350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076708.389352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076708.389354:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076708.389364:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076708.389379:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076708.389385:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076708/real 1769076708] req@ffff969e06a62680 x1855007923771392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076708.389392:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923771392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076708.389398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076708.389400:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076708.389402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076708.389413:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076708.389423:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076708.389429:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076708/real 1769076708] req@ffff969e06a61500 x1855007923771520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076708.389452:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923771520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076708.389458:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076708.389460:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076708.389462:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076708.708688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076709.412906:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076709.732689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076710.436770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076710.756678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076711.460785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076711.780711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076712.484792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076712.804761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076713.508734:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076713.508747:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076713.508753:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769076713.508763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076713.508765:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3402 00000100:00080000:0.0:1769076713.508780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076713.508788:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076713.508832:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923771648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076713.508860:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076713.508863:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076713.508864:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076713.508868:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3402 00000100:00080000:0.0:1769076713.508892:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076713.508900:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076713.508902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923771648:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076713.508912:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076713.508929:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923771776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076713.508940:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076713.508944:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076713.508946:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076713.508950:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3402 00000100:00080000:0.0:1769076713.508962:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076713.508965:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076713.508969:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076713.508975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076713.508989:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923771904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076713.508999:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076713.509002:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076713.509004:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076713.509009:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3402 00000100:00080000:0.0:1769076713.509013:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076713.509017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076713/real 1769076713] req@ffff969e06a61180 x1855007923771648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076713.509018:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076713.509032:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076713.509053:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923772032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076713.509057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923771648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076713.509104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076713.509121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076713.509123:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076713.509128:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076713.509133:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076713.509134:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923771648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076713.509138:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3402 00000100:00080000:0.0:1769076713.509142:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076713.509146:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076713.509152:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076713.509162:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923772160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076713.509175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076713.509200:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076713.509210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076713/real 1769076713] req@ffff969e06a60380 x1855007923771776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076713.509218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923771776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076713.509225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076713.509227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076713.509230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076713.509235:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076713.509245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076713.509251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076713/real 1769076713] req@ffff969e06a60a80 x1855007923771904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076713.509272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923771904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076713.509277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076713.509279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076713.509282:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923771904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076713.509286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076713.509299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076713.509305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076713/real 1769076713] req@ffff969e06a60700 x1855007923772032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076713.509313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923772032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076713.509318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076713.509320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076713.509321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076713.509333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076713.509342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076713.509369:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076713/real 1769076713] req@ffff969e06a63800 x1855007923772160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076713.509377:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923772160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076713.509398:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076713.509402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076713.509406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076713.828656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076714.532751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076714.852737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076715.556868:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076715.876749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076716.580781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076716.900755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076717.604828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076717.924736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076718.628785:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076718.628799:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076718.628805:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076718.628817:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3407 00000100:00080000:2.0:1769076718.628832:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076718.628842:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076718.628870:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076718.628906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923772288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076718.628945:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076718.628950:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076718.628953:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076718.628958:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3407 00000100:00080000:2.0:1769076718.628982:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076718.628989:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076718.629002:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076718.629007:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923772288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076718.629026:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923772416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076718.629044:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076718.629048:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076718.629051:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076718.629057:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3407 00000100:00080000:2.0:1769076718.629063:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076718.629068:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076718.629073:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076718.629088:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923772544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076718.629102:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076718.629105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076718.629107:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076718.629115:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076718.629122:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3407 00000100:00080000:2.0:1769076718.629129:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076718.629133:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076718.629141:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076718.629156:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923772672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076718.629188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076718/real 1769076718] req@ffff969e0a46ed80 x1855007923772288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076718.629199:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076718.629211:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076718.629217:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076718.629230:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3407 00000100:00080000:2.0:1769076718.629240:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076718.629247:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076718.629255:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076718.629313:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923772800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076718.629316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923772288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076718.629354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076718.629374:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076718.629383:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923772288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076718.629437:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076718.629484:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076718.629510:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076718/real 1769076718] req@ffff969e0a46d880 x1855007923772416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076718.629538:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923772416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076718.629563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076718.629568:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076718.629573:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076718.629582:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076718.629646:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076718.629662:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076718/real 1769076718] req@ffff969e0a46ea00 x1855007923772544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076718.629677:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923772544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076718.629699:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076718.629703:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076718.629707:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076718.629714:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076718.629734:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076718.629745:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076718/real 1769076718] req@ffff969e0a46ca80 x1855007923772672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076718.629759:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923772672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076718.629769:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076718.629773:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076718.629787:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076718.629795:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076718.629819:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076718.629832:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076718/real 1769076718] req@ffff969e0a46df80 x1855007923772800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076718.629846:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923772800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076718.629856:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076718.629859:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076718.629863:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923772800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076718.948716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076719.652799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076719.972762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076720.676854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076720.996759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076721.700878:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076722.020771:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076722.724791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076723.044720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076723.748787:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076723.748806:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076723.748814:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769076723.748830:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076723.748834:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3412 00000100:00080000:0.0:1769076723.748855:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076723.748866:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076723.748923:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923772928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076723.748960:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076723.748965:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076723.748967:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076723.748973:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3412 00000100:00080000:0.0:1769076723.748978:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076723.748983:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076723.749017:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076723.749031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923772928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076723.749050:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923773056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076723.749089:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076723.749096:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076723.749101:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076723.749109:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3412 00000100:00080000:0.0:1769076723.749120:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076723.749129:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076723.749137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769076723.749147:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076723.749211:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923773184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076723.749227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076723/real 1769076723] req@ffff969e06a60000 x1855007923772928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076723.749231:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076723.749250:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076723.749257:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076723.749280:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3412 00000100:00080000:0.0:1769076723.749288:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076723.749293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923772928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076723.749301:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076723.749315:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076723.749323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076723.749334:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076723.749340:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923773312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076723.749367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923772928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076723.749368:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076723.749385:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076723.749403:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076723.749410:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3412 00000100:00080000:0.0:1769076723.749416:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076723.749420:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773056:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769076723.749430:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076723.749441:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076723.749462:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923773440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076723.749478:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076723.749504:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076723/real 1769076723] req@ffff969e06a62d80 x1855007923773056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076723.749522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923773056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076723.749534:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076723.749538:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076723.749542:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076723.749550:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076723.749570:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076723.749581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076723/real 1769076723] req@ffff969e06a63800 x1855007923773184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076723.749631:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923773184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076723.749642:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076723.749646:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076723.749650:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076723.749657:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076723.749691:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076723.749717:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076723/real 1769076723] req@ffff969e06a60700 x1855007923773312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076723.749733:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923773312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076723.749744:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076723.749748:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076723.749752:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076723.749773:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076723.749796:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076723.749813:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076723/real 1769076723] req@ffff969e06a60a80 x1855007923773440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076723.749837:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923773440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076723.749854:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076723.749860:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076723.749877:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076724.068748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076724.772873:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076725.092726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076725.797697:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076726.116779:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076726.820824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076727.140703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076727.844806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076728.164753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076728.869202:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076728.869222:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076728.869230:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076728.869247:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3418 00000100:00080000:3.0:1769076728.869265:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076728.869279:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076728.869343:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923773568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076728.869417:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076728.869427:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076728.869431:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076728.869441:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3418 00000100:00080000:3.0:1769076728.869451:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076728.869461:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076728.869480:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076728.869509:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923773696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076728.869563:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076728.869572:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923773568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076728.869574:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076728.869585:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076728.869681:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3418 00000100:00080000:3.0:1769076728.869695:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076728.869704:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076728.869717:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076728.869733:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076728.869755:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923773824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076728.869757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076728.869798:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076728.869806:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076728.869810:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076728.869820:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3418 00000100:00080000:3.0:1769076728.869828:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076728.869835:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076728.869846:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076728.869852:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076728/real 1769076728] req@ffff969e0a7fd180 x1855007923773568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076728.869878:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923773952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076728.869903:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923773568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076728.869922:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076728.869935:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076728.869939:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076728.869951:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076728.869954:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3418 00000100:00080000:1.0:1769076728.869967:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076728.869986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923773568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076728.869987:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076728.870006:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076728.870018:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076728.870052:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923774080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076728.870068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076728.870162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076728.870201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076728/real 1769076728] req@ffff969e0a7fdf80 x1855007923773696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076728.870245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923773696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076728.870262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076728.870270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076728.870277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076728.870291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076728.870333:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076728.870357:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076728/real 1769076728] req@ffff969e0a7fc380 x1855007923773824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076728.870384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923773824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076728.870421:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076728.870429:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076728.870436:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076728.870450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076728.870498:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076728.870522:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076728/real 1769076728] req@ffff969e0a7fe300 x1855007923773952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076728.870549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923773952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076728.870560:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076728.870565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076728.870569:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923773952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076728.870625:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076728.870660:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076728.870677:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076728/real 1769076728] req@ffff969e0a7fca80 x1855007923774080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076728.870693:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923774080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076728.870704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076728.870708:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076728.870726:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076729.188714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076729.893041:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076730.212682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076730.917186:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076731.236674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076731.940731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076732.260644:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076732.964760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076733.284676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076733.988710:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076733.988721:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076733.988726:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076733.988737:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3423 00010000:00010000:2.0:1769076733.988737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076733.988753:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076733.988761:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076733.988804:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923774208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076733.988831:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076733.988834:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076733.988836:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076733.988839:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3423 00000100:00080000:0.0:1769076733.988842:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076733.988846:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076733.988856:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076733.988866:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923774336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076733.988880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923774208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076733.988911:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076733.988914:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076733.988916:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076733.988922:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3423 00000100:00080000:0.0:1769076733.988927:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076733.988930:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076733.988935:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076733.988946:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923774464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076733.988953:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076733.988963:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076733.988966:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076733.988968:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076733.988973:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3423 00000100:00080000:0.0:1769076733.988976:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076733.988980:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076733.988985:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076733.988993:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923774592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076733.989013:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076733.989018:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076733.989022:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076733.989023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076733/real 1769076733] req@ffff969e06a60380 x1855007923774208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076733.989028:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3423 00000100:00080000:0.0:1769076733.989043:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076733.989052:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076733.989057:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076733.989066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923774208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076733.989071:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923774720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076733.989105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076733.989113:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076733.989119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923774208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076733.989149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076733.989174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076733.989185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076733/real 1769076733] req@ffff969e06a61180 x1855007923774336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076733.989194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923774336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076733.989200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076733.989203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076733.989218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076733.989224:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076733.989239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076733.989246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076733/real 1769076733] req@ffff969e06a61500 x1855007923774464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076733.989255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923774464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076733.989260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076733.989262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076733.989264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076733.989269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076733.989280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076733.989286:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076733/real 1769076733] req@ffff969e06a62680 x1855007923774592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076733.989293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923774592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076733.989300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076733.989304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076733.989307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076733.989313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076733.989332:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076733.989351:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076733/real 1769076733] req@ffff969e06a61f80 x1855007923774720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076733.989364:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923774720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076733.989370:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076733.989372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076733.989375:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076734.308675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076735.012780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076735.332652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076736.036763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076736.356751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076737.060849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076737.380748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076738.084754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076738.404702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076739.108834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076739.108856:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076739.108863:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076739.108868:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076739.108886:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3428 00000100:00080000:2.0:1769076739.108900:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076739.108970:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076739.109010:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923774848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076739.109041:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076739.109044:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076739.109045:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076739.109049:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3428 00000100:00080000:2.0:1769076739.109052:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076739.109055:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076739.109066:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076739.109077:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923774976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076739.109087:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076739.109093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923774848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076739.109093:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076739.109100:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076739.109115:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3428 00000100:00080000:2.0:1769076739.109117:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076739.109120:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076739.109126:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076739.109141:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923775104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076739.109148:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076739.109152:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076739.109165:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076739.109171:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3428 00000100:00080000:2.0:1769076739.109175:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076739.109182:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076739.109186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769076739.109195:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076739.109208:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923775232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076739.109219:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076739.109222:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076739.109224:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076739.109244:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3428 00000100:00100000:1.0:1769076739.109246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076739/real 1769076739] req@ffff969e0a46e300 x1855007923774848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076739.109248:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076739.109261:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076739.109281:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076739.109295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923774848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076739.109310:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923775360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076739.109312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076739.109325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076739.109343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923774848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076739.109384:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076739.109411:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076739.109445:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076739/real 1769076739] req@ffff969e0a46ce00 x1855007923774976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076739.109460:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923774976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076739.109469:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076739.109473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076739.109478:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923774976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076739.109488:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076739.109521:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076739.109539:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076739/real 1769076739] req@ffff969e0a46df80 x1855007923775104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076739.109562:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923775104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076739.109576:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076739.109582:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076739.109588:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076739.109598:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076739.109672:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076739.109690:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076739/real 1769076739] req@ffff969e0a46ca80 x1855007923775232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076739.109709:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923775232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076739.109751:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076739.109758:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076739.109764:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076739.109777:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076739.109809:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076739.109822:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076739/real 1769076739] req@ffff969e0a46ea00 x1855007923775360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076739.109834:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923775360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076739.109844:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076739.109848:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076739.109851:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076739.428659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076740.132772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076740.452699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076741.156973:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076741.476748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076742.180908:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076742.500719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076743.204760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076743.524706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076744.228747:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076744.228759:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076744.228764:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076744.228775:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3433 00000100:00080000:2.0:1769076744.228789:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076744.228820:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076744.228845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076744.228875:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923775488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076744.228903:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076744.228906:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076744.228907:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076744.228911:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3433 00000100:00080000:2.0:1769076744.228915:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076744.228919:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076744.228929:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076744.228944:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923775488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076744.228945:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923775616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076744.228967:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076744.228969:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076744.228971:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076744.228976:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3433 00000100:00080000:2.0:1769076744.228981:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076744.228985:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076744.228989:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076744.228998:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923775744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076744.229006:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076744.229009:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076744.229018:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076744.229022:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3433 00000800:00000100:1.0:1769076744.229033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076744.229036:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076744.229042:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076744.229048:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076744.229062:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923775872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076744.229077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076744/real 1769076744] req@ffff969e0a46d880 x1855007923775488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076744.229083:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076744.229098:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076744.229104:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076744.229110:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3433 00000100:00080000:2.0:1769076744.229116:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076744.229121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923775488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076744.229125:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076744.229130:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076744.229144:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923776000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076744.229153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076744.229163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076744.229170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923775488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076744.229196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076744.229218:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076744.229226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076744/real 1769076744] req@ffff969e0a46ed80 x1855007923775616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076744.229235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923775616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076744.229240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076744.229243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076744.229245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076744.229249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076744.229259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076744.229264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076744/real 1769076744] req@ffff969e0a46f480 x1855007923775744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076744.229271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923775744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076744.229277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076744.229278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076744.229280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076744.229290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076744.229304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076744.229311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076744/real 1769076744] req@ffff969e0a46d500 x1855007923775872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076744.229318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923775872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076744.229324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076744.229326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076744.229327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923775872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076744.229331:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076744.229341:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076744.229346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076744/real 1769076744] req@ffff969e0a46c380 x1855007923776000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076744.229353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923776000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076744.229358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076744.229360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076744.229362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076744.548699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076745.252764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076745.572739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076746.276757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076746.596695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076747.300741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076747.620708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076748.324798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076748.644669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076749.348690:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769076749.348696:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076749.348702:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076749.348708:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076749.348718:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3438 00000100:00080000:0.0:1769076749.348729:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076749.348736:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076749.348804:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923776128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076749.348829:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076749.348833:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076749.348835:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076749.348840:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3438 00000100:00080000:0.0:1769076749.348845:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076749.348850:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076749.348860:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923776128:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076749.348861:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076749.348878:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923776256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076749.348890:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076749.348909:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076749.348912:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076749.348916:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3438 00000100:00080000:0.0:1769076749.348921:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076749.348923:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076749.348928:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076749.348935:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076749.348950:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923776384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076749.348958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076749/real 1769076749] req@ffff969e06a63100 x1855007923776128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076749.348967:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076749.348972:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076749.348974:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076749.348978:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923776128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076749.348982:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3438 00000100:00080000:0.0:1769076749.348990:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076749.348994:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076749.348996:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076749.349001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:0.0:1769076749.349003:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076749.349005:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923776128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076749.349018:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923776512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076749.349029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776256:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769076749.349032:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076749.349037:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076749.349040:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076749.349045:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3438 00000800:00000100:1.0:1769076749.349049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076749.349052:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076749.349059:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076749.349062:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076749/real 1769076749] req@ffff969e06a61880 x1855007923776256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076749.349075:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076749.349077:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923776256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076749.349087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076749.349092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076749.349093:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923776640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076749.349096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076749.349106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076749.349125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076749.349135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076749/real 1769076749] req@ffff969e06a61f80 x1855007923776384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076749.349152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923776384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076749.349157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076749.349159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076749.349161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076749.349172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076749.349185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076749.349192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076749/real 1769076749] req@ffff969e06a62680 x1855007923776512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076749.349199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923776512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076749.349205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076749.349206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076749.349208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076749.349212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076749.349221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076749.349226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076749/real 1769076749] req@ffff969e06a61500 x1855007923776640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076749.349233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923776640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076749.349247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076749.349250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076749.349252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076749.668682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076750.372860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076750.692640:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076751.396787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076751.716753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076752.420779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076752.740724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076753.444768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076753.764728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076754.468746:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076754.468754:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076754.468758:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076754.468766:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3443 00000100:00080000:1.0:1769076754.468776:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076754.468783:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076754.468818:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923776768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076754.468848:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076754.468852:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:2.0:1769076754.468852:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076754.468854:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076754.468863:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3443 00000100:00080000:1.0:1769076754.468866:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076754.468870:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076754.468879:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076754.468884:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923776768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076754.468891:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923776896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076754.468900:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076754.468915:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076754.468916:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076754.468920:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3443 00000100:00080000:1.0:1769076754.468930:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076754.468934:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076754.468938:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076754.468943:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076754.468949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923777024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076754.468958:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076754.468961:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076754.468962:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076754.468965:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3443 00000100:00080000:1.0:1769076754.468968:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076754.468971:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076754.468975:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076754.468983:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076754/real 1769076754] req@ffff969e10b60380 x1855007923776768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076754.468984:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923777152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076754.469005:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076754.469011:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076754.469013:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076754.469016:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3443 00000100:00080000:1.0:1769076754.469019:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076754.469021:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076754.469026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923776768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769076754.469031:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769076754.469043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769076754.469048:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923777280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076754.469051:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076754.469059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923776768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076754.469085:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076754.469104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076754.469113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076754/real 1769076754] req@ffff969e10b61180 x1855007923776896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076754.469121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923776896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076754.469127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076754.469129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076754.469131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923776896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076754.469135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076754.469156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076754.469163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076754/real 1769076754] req@ffff969e10b60000 x1855007923777024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076754.469171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923777024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076754.469176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076754.469178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076754.469179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076754.469183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076754.469194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076754.469199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076754/real 1769076754] req@ffff969e10b62d80 x1855007923777152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076754.469206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923777152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076754.469212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076754.469213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076754.469215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769076754.469219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076754.469228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076754.469234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076754/real 1769076754] req@ffff969e10b60700 x1855007923777280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076754.469251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923777280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076754.469257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076754.469259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076754.469261:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076754.788715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076755.492781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076755.812776:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076756.516818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076756.836724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076757.540788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076757.860738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076758.564791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076758.884707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076759.588723:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076759.588732:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076759.588736:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076759.588744:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3448 00000100:00080000:2.0:1769076759.588753:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076759.588759:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076759.588792:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923777408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769076759.588796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076759.588820:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076759.588823:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076759.588824:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076759.588828:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3448 00000100:00080000:2.0:1769076759.588830:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076759.588832:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076759.588839:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076759.588849:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923777536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076759.588878:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076759.588884:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076759.588886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923777408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076759.588887:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076759.588898:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3448 00000100:00080000:2.0:1769076759.588902:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076759.588905:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076759.588909:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076759.588928:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923777664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076759.588935:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076759.588937:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076759.588938:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076759.588942:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3448 00000100:00080000:2.0:1769076759.588944:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076759.588947:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076759.588950:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076759.588956:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923777792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076759.588962:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076759.588965:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076759.588967:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076759.588971:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3448 00000100:00080000:2.0:1769076759.588979:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769076759.588985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076759.588985:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076759.588998:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076759.589010:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923777920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076759.589043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076759/real 1769076759] req@ffff969e0a46f800 x1855007923777408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076759.589060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923777408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076759.589070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076759.589074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076759.589077:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923777408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076759.589104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076759.589121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076759.589129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076759/real 1769076759] req@ffff969e0a46f100 x1855007923777536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076759.589137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923777536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076759.589161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076759.589164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076759.589166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076759.589171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076759.589186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076759.589192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076759/real 1769076759] req@ffff969e0a46c380 x1855007923777664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076759.589200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923777664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076759.589206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076759.589207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076759.589209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076759.589213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076759.589225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076759.589235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076759/real 1769076759] req@ffff969e0a46d500 x1855007923777792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076759.589247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923777792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076759.589256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076759.589260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076759.589272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769076759.589280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076759.589301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076759.589310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076759/real 1769076759] req@ffff969e0a46f480 x1855007923777920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076759.589318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923777920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076759.589325:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076759.589327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076759.589329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923777920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076759.908666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076760.612790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076760.932682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076761.636727:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076761.956688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076762.660720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076762.980706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076763.684763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076764.004695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076764.708666:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076764.708674:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076764.708678:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076764.708689:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3453 00000100:00080000:2.0:1769076764.708701:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076764.708707:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076764.708725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076764.708746:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923778048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076764.708778:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076764.708782:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076764.708784:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076764.708788:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3453 00000100:00080000:2.0:1769076764.708793:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076764.708798:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076764.708823:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076764.708832:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923778048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076764.708843:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923778176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076764.708855:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076764.708858:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076764.708860:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076764.708863:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3453 00000100:00080000:2.0:1769076764.708866:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076764.708869:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076764.708873:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076764.708881:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923778304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076764.708887:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076764.708891:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076764.708892:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076764.708894:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3453 00000100:00080000:2.0:1769076764.708896:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076764.708898:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076764.708901:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076764.708909:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923778432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076764.708915:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076764.708917:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076764.708924:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076764.708927:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3453 00000100:00080000:2.0:1769076764.708930:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076764.708932:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076764.708936:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076764.708946:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923778560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076764.708975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076764.709006:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076764.709026:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769076764.709052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076765.028719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076765.732998:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076766.052757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076766.757048:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076767.076703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000400:00000100:0.0:1769076767.781136:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769076767.781198:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769076767.781217:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00010000:00010000:3.0:1769076767.781232:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:0.0:1769076767.781269:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076767.781276:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076767.781301:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076767.781313:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076767.781320:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076764/real 1769076767] req@ffff969e0a46d880 x1855007923778176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076767.781378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923778176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076767.781392:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076767.781399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076767.781404:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076767.781415:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076764/real 1769076767] req@ffff969e0a46ea00 x1855007923778304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076767.781428:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923778304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076767.781484:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076767.781488:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076767.781493:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076767.781513:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076764/real 1769076767] req@ffff969e0a46ca80 x1855007923778432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076767.781529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923778432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076767.781574:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076767.781579:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076767.781584:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076767.781642:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076764/real 1769076767] req@ffff969e0a46df80 x1855007923778560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076767.781658:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923778560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076767.781669:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076767.781674:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076767.781696:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076767.781709:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076764/real 1769076767] req@ffff969e0a46ed80 x1855007923778048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076767.781731:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923778048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076767.781741:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076767.781745:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076767.781749:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923778048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076768.100711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076768.804969:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076769.124676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076769.828742:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076769.828751:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076769.828756:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076769.828767:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3459 00000100:00080000:3.0:1769076769.828780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076769.828789:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076769.828832:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923778688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076769.828855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076769.828873:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076769.828878:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076769.828912:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076769.828919:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3459 00000100:00080000:3.0:1769076769.828925:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076769.828930:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076769.828941:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923778688:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076769.828950:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076769.828970:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923778816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076769.828983:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076769.828987:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076769.828990:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076769.828994:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3459 00000100:00080000:3.0:1769076769.828998:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076769.829003:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076769.829008:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076769.829021:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923778944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076769.829033:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076769.829036:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076769.829041:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076769.829044:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076769.829050:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3459 00000100:00080000:3.0:1769076769.829069:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076769.829074:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076769.829087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076769/real 1769076769] req@ffff969e0a7fd880 x1855007923778688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076769.829092:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076769.829119:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923779072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076769.829131:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076769.829134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923778688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076769.829136:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076769.829140:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076769.829150:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3459 00000100:00080000:1.0:1769076769.829159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076769.829160:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076769.829170:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076769.829173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:3.0:1769076769.829179:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076769.829181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923778688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076769.829196:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923779200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076769.829217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076769.829251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076769.829269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076769/real 1769076769] req@ffff969e0a7fea00 x1855007923778816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076769.829283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923778816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076769.829293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076769.829296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076769.829300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076769.829307:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076769.829331:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076769.829344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076769/real 1769076769] req@ffff969e0a7fc700 x1855007923778944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076769.829359:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923778944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076769.829384:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076769.829388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076769.829392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923778944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076769.829412:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076769.829445:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076769.829459:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076769/real 1769076769] req@ffff969e0a7fce00 x1855007923779072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076769.829480:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923779072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076769.829490:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076769.829493:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076769.829497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076769.829503:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076769.829524:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076769.829537:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076769/real 1769076769] req@ffff969e0a7fe680 x1855007923779200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076769.829550:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923779200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076769.829559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076769.829562:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076769.829574:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076770.148686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076770.853014:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076771.172682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076771.876794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076772.196693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076772.900790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076773.220707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076773.924741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076774.244700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076774.948650:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076774.948658:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076774.948662:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076774.948670:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3464 00000100:00080000:3.0:1769076774.948679:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076774.948685:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076774.948720:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923779328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076774.948746:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769076774.948747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076774.948750:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076774.948753:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076774.948757:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3464 00000100:00080000:3.0:1769076774.948762:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076774.948779:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923779328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076774.948780:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076774.948793:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076774.948808:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923779456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076774.948817:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076774.948820:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076774.948821:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076774.948824:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3464 00000100:00080000:3.0:1769076774.948828:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076774.948831:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076774.948835:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076774.948843:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076774.948845:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923779584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076774.948853:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076774.948856:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076774.948858:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076774.948860:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3464 00000100:00080000:3.0:1769076774.948863:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076774.948866:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076774.948870:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076774.948874:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076774/real 1769076774] req@ffff969e0a7fed80 x1855007923779328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076774.948879:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923779712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076774.948896:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076774.948902:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923779328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076774.948909:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076774.948913:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076774.948920:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3464 00000100:00080000:1.0:1769076774.948921:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076774.948924:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076774.948928:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076774.948932:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076774.948936:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923779328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769076774.948939:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076774.948950:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923779840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076774.948958:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076774.948976:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076774.948985:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076774/real 1769076774] req@ffff969e0a7fe680 x1855007923779456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076774.948993:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923779456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076774.948998:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076774.949001:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076774.949003:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076774.949007:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076774.949016:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076774.949031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076774/real 1769076774] req@ffff969e0a7fce00 x1855007923779584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076774.949039:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923779584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076774.949044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076774.949046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076774.949048:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076774.949052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076774.949064:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076774.949069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076774/real 1769076774] req@ffff969e0a7fc700 x1855007923779712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076774.949076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923779712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076774.949082:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076774.949083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076774.949085:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076774.949104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076774.949116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076774.949122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076774/real 1769076774] req@ffff969e0a7fea00 x1855007923779840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076774.949129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923779840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076774.949139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076774.949142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076774.949144:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923779840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076775.268686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076775.972766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076776.292689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076776.996813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076777.316660:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076778.020773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076778.340716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076779.044857:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076779.364738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076780.068731:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076780.068747:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076780.068755:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769076780.068783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076780.068786:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3469 00000100:00080000:2.0:1769076780.068802:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076780.068809:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076780.068866:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923779968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076780.068904:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076780.068919:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076780.068922:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076780.068927:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3469 00000100:00080000:2.0:1769076780.068932:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076780.068939:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076780.068962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923779968:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076780.068968:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076780.068987:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923780096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076780.069007:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076780.069012:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076780.069014:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076780.069020:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3469 00000100:00080000:2.0:1769076780.069026:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076780.069051:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076780.069058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769076780.069058:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076780.069080:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923780224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076780.069095:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076780.069099:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076780.069101:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076780.069109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076780/real 1769076780] req@ffff969e0a46ce00 x1855007923779968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076780.069112:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3469 00000100:00080000:2.0:1769076780.069127:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076780.069137:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076780.069145:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076780.069146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923779968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076780.069176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076780.069182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076780.069184:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923780352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076780.069190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923779968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076780.069216:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076780.069221:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076780.069224:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076780.069256:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3469 00000100:00080000:2.0:1769076780.069263:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076780.069270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780096:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076780.069274:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076780.069291:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076780.069316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076780.069317:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923780480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076780.069332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076780/real 1769076780] req@ffff969e0a46e300 x1855007923780096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076780.069345:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923780096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076780.069353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076780.069357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076780.069360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076780.069379:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076780.069395:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076780.069403:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076780/real 1769076780] req@ffff969e0a46ed80 x1855007923780224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076780.069411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923780224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076780.069417:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076780.069420:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076780.069422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076780.069426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076780.069437:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076780.069443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076780/real 1769076780] req@ffff969e0a46df80 x1855007923780352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076780.069450:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923780352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076780.069457:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076780.069458:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076780.069460:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076780.069474:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076780.069484:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076780.069490:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076780/real 1769076780] req@ffff969e0a46ca80 x1855007923780480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076780.069506:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923780480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076780.069513:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076780.069516:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076780.069518:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076780.388749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076781.092875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076781.412666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076782.116778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076782.436689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076783.140829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076783.460671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076784.164790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076784.484714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769076785.188733:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076785.188751:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076785.188757:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076785.188769:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3474 00010000:00010000:3.0:1769076785.188777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076785.188783:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076785.188793:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076785.188843:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923780608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076785.188873:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076785.188877:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076785.188879:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076785.188884:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3474 00000100:00080000:1.0:1769076785.188888:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076785.188892:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076785.188904:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076785.188917:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923780736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076785.188929:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923780608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769076785.188958:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076785.188962:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076785.188964:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076785.188970:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3474 00000100:00080000:1.0:1769076785.188975:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076785.188979:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076785.188986:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076785.189000:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923780864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076785.189011:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076785.189015:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769076785.189017:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769076785.189025:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769076785.189031:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3474 00000100:00080000:1.0:1769076785.189037:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076785.189041:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769076785.189047:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076785.189060:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923780992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076785.189070:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076785.189075:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076785/real 1769076785] req@ffff969e10b62680 x1855007923780608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076785.189076:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076785.189082:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076785.189095:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3474 00000100:00080000:1.0:1769076785.189110:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076785.189112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923780608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076785.189129:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769076785.189130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076785.189173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076785.189179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923780608:192.168.204.143@tcp:250:kworker.0 00000008:00100000:1.0:1769076785.189179:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076785.189204:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923781120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076785.189221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076785.189255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076785.189269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076785/real 1769076785] req@ffff969e10b62a00 x1855007923780736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076785.189281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923780736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076785.189290:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076785.189294:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076785.189297:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076785.189304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076785.189319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076785.189342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076785/real 1769076785] req@ffff969e10b60a80 x1855007923780864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076785.189354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923780864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076785.189362:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076785.189365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076785.189368:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076785.189374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076785.189394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076785.189410:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076785/real 1769076785] req@ffff969e10b61f80 x1855007923780992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076785.189428:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923780992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076785.189442:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076785.189447:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076785.189453:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923780992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769076785.189480:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076785.189508:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076785.189520:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076785/real 1769076785] req@ffff969e10b61500 x1855007923781120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076785.189536:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923781120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076785.189564:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076785.189571:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076785.189576:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076785.508705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076786.212763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076786.532688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076787.236771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076787.556735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076788.260821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076788.580689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076789.284747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076789.604670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076790.308852:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076790.308865:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076790.308871:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076790.308895:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3479 00000100:00080000:2.0:1769076790.308909:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076790.308917:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076790.308918:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076790.308980:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923781248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076790.309008:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076790.309012:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076790.309014:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076790.309017:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3479 00000100:00080000:2.0:1769076790.309019:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076790.309022:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076790.309033:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076790.309043:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923781376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076790.309053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923781248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076790.309100:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076790.309103:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076790.309105:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076790.309110:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3479 00000100:00080000:2.0:1769076790.309114:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076790.309117:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076790.309123:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076790.309150:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923781504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076790.309163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076790.309174:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076790.309178:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076790.309181:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076790.309185:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3479 00000100:00080000:2.0:1769076790.309190:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076790.309194:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076790.309200:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076790.309211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076790/real 1769076790] req@ffff969e0a46ea00 x1855007923781248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076790.309216:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923781632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076790.309262:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076790.309265:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076790.309267:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076790.309272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923781248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076790.309277:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3479 00000100:00080000:0.0:1769076790.309287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076790.309289:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076790.309312:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769076790.309323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769076790.309326:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076790.309332:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923781248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076790.309343:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923781760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076790.309365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076790.309390:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076790.309400:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076790/real 1769076790] req@ffff969e0a46d880 x1855007923781376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076790.309435:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923781376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076790.309442:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076790.309445:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076790.309447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076790.309466:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769076790.309500:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076790.309510:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076790/real 1769076790] req@ffff969e0a46f480 x1855007923781504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076790.309518:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923781504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076790.309524:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076790.309526:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076790.309528:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076790.309548:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076790.309562:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076790.309569:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076790/real 1769076790] req@ffff969e0a46d500 x1855007923781632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076790.309576:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923781632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076790.309582:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076790.309587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076790.309591:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769076790.309595:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769076790.309612:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076790.309646:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076790/real 1769076790] req@ffff969e0a46c380 x1855007923781760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076790.309677:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923781760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076790.309683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076790.309686:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076790.309688:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923781760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076790.628706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076791.332922:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076791.652694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076792.357175:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076792.676685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076793.380961:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076793.700697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076794.405668:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076794.724710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076795.428813:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076795.428830:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076795.428837:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076795.428851:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3484 00000100:00080000:2.0:1769076795.428890:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076795.428902:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076795.428959:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923781888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076795.429024:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076795.429032:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076795.429037:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076795.429044:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3484 00000100:00080000:2.0:1769076795.429050:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076795.429057:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076795.429080:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076795.429130:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923782016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076795.429150:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076795.429156:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076795.429160:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076795.429168:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3484 00000100:00080000:2.0:1769076795.429174:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076795.429181:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076795.429190:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076795.429211:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923782144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076795.429226:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076795.429231:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076795.429234:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076795.429240:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3484 00000100:00080000:2.0:1769076795.429246:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076795.429251:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076795.429259:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076795.429275:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923782272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076795.429289:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076795.429295:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076795.429298:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076795.429315:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3484 00000100:00080000:2.0:1769076795.429325:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076795.429331:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076795.429340:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076795.429359:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923782400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076795.429774:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923781888:192.168.204.143@tcp:250:kworker.0 00010000:00010000:3.0:1769076795.429776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1769076795.429893:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076795.429955:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076795/real 1769076795] req@ffff969e0a46f100 x1855007923781888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076795.430020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923781888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076795.430039:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076795.430046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076795.430053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923781888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076795.430066:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076795.430107:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076795.430148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076795/real 1769076795] req@ffff969e0a46f800 x1855007923782016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076795.430175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923782016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076795.430191:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076795.430214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076795.430233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076795.430245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076795.430284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076795.430304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076795/real 1769076795] req@ffff969e0a46c380 x1855007923782144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076795.430326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923782144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076795.430339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076795.430344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076795.430350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076795.430361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076795.430395:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076795.430414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076795/real 1769076795] req@ffff969e0a46d500 x1855007923782272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076795.430427:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923782272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076795.430437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076795.430440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076795.430443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076795.430450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076795.430490:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076795.430543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076795/real 1769076795] req@ffff969e0a46f480 x1855007923782400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076795.430566:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923782400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076795.430582:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076795.430622:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076795.430630:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076795.748708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076796.453824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076796.772739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076797.477182:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076797.796742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076798.500874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076798.820777:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076799.524787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076799.844712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076800.548711:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769076800.548715:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076800.548721:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076800.548725:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076800.548733:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3489 00000100:00080000:0.0:1769076800.548742:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076800.548748:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076800.548780:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923782528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076800.548801:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076800.548803:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076800.548805:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076800.548808:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3489 00000100:00080000:0.0:1769076800.548810:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076800.548813:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076800.548820:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076800.548830:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923782656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076800.548835:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923782528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076800.548860:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076800.548863:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076800.548865:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076800.548869:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3489 00000100:00080000:0.0:1769076800.548881:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076800.548905:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076800.548910:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076800.548912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076800.548923:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923782784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076800.548931:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076800.548934:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076800.548935:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076800.548939:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3489 00000100:00080000:0.0:1769076800.548943:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076800.548948:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076800.548948:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076800/real 1769076800] req@ffff969e06a61180 x1855007923782528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076800.548954:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076800.548967:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923782912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076800.548973:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923782528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076800.548978:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076800.548986:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076800.548989:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076800.548995:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3489 00000100:00080000:1.0:1769076800.548997:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076800.549006:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076800.549009:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076800.549013:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076800.549018:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076800.549021:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923782528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076800.549028:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923783040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076800.549067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076800.549092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076800.549101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076800/real 1769076800] req@ffff969e06a60380 x1855007923782656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076800.549110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923782656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076800.549126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076800.549129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076800.549131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076800.549135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076800.549149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076800.549156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076800/real 1769076800] req@ffff969e06a60a80 x1855007923782784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076800.549163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923782784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076800.549168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076800.549170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076800.549172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076800.549176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076800.549186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076800.549191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076800/real 1769076800] req@ffff969e06a60700 x1855007923782912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076800.549199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923782912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076800.549203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076800.549205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076800.549212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923782912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076800.549216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076800.549229:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076800.549235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076800/real 1769076800] req@ffff969e06a63800 x1855007923783040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076800.549242:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923783040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076800.549248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076800.549249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076800.549251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076800.868670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076801.572739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076801.892727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076802.596738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076802.916637:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076803.620895:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076803.940735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076804.644797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076804.964728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076805.668708:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076805.668718:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076805.668723:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076805.668732:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3494 00000100:00080000:0.0:1769076805.668743:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076805.668750:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076805.668773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076805.668789:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923783168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076805.668831:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076805.668835:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076805.668837:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076805.668853:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3494 00000100:00080000:0.0:1769076805.668857:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076805.668861:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076805.668871:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076805.668878:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923783168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076805.668888:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923783296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076805.668899:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076805.668920:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076805.668922:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076805.668926:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3494 00000100:00080000:0.0:1769076805.668931:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076805.668936:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076805.668940:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076805.668947:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076805.668955:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923783424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076805.668965:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076805.668968:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076805.668970:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076805.668983:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3494 00000100:00080000:0.0:1769076805.668987:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076805.668991:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076805.668996:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076805.669000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076805/real 1769076805] req@ffff969e06a62d80 x1855007923783168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076805.669009:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923783552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076805.669021:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076805.669024:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076805.669027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923783168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076805.669028:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076805.669038:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3494 00000100:00080000:0.0:1769076805.669045:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076805.669051:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076805.669052:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076805.669069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:0.0:1769076805.669078:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076805.669078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923783168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076805.669095:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923783680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076805.669110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076805.669132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076805.669166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076805/real 1769076805] req@ffff969e06a60000 x1855007923783296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076805.669177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923783296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076805.669184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076805.669187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076805.669190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076805.669194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076805.669209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076805.669217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076805/real 1769076805] req@ffff969e06a63800 x1855007923783424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076805.669226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923783424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076805.669232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076805.669234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076805.669237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076805.669241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076805.669253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076805.669260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076805/real 1769076805] req@ffff969e06a60700 x1855007923783552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076805.669269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923783552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076805.669282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076805.669285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076805.669287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076805.669301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076805.669329:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076805.669344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076805/real 1769076805] req@ffff969e06a60a80 x1855007923783680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076805.669354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923783680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076805.669361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076805.669363:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076805.669366:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076805.988724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076806.692734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076807.012690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076807.716799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076808.036726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076808.740879:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076809.060694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076809.764905:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076810.084779:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076810.788763:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076810.788773:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076810.788778:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076810.788788:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3499 00000100:00080000:0.0:1769076810.788799:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076810.788806:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076810.788845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076810.788846:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923783808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076810.788873:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076810.788876:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076810.788878:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076810.788882:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3499 00000100:00080000:0.0:1769076810.788884:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076810.788887:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076810.788896:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076810.788906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923783936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076810.788916:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076810.788921:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076810.788925:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076810.788925:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923783808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076810.788932:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3499 00000100:00080000:0.0:1769076810.788952:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076810.788956:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076810.788961:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076810.788974:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923784064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076810.788982:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076810.788985:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076810.788987:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076810.788991:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3499 00000100:00080000:0.0:1769076810.788994:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076810.788997:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076810.789000:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076810.789010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076810.789012:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923784192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076810.789023:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076810.789026:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076810.789028:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076810.789031:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3499 00000100:00080000:0.0:1769076810.789035:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076810.789039:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076810.789043:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076810.789051:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923784320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076810.789066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076810/real 1769076810] req@ffff969e06a60380 x1855007923783808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076810.789109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923783808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076810.789124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076810.789131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076810.789137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923783808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076810.789172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076810.789203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076810.789221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076810/real 1769076810] req@ffff969e06a61180 x1855007923783936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076810.789239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923783936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076810.789251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076810.789256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076810.789261:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923783936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076810.789270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076810.789297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076810.789312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076810/real 1769076810] req@ffff969e06a61500 x1855007923784064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076810.789329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923784064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076810.789340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076810.789345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076810.789350:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076810.789358:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076810.789384:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076810.789399:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076810/real 1769076810] req@ffff969e06a62680 x1855007923784192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076810.789425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923784192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076810.789438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076810.789443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076810.789447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076810.789455:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076810.789493:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076810.789508:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076810/real 1769076810] req@ffff969e06a61f80 x1855007923784320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076810.789524:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923784320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076810.789536:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076810.789540:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076810.789545:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076811.108696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076811.812991:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076812.132749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076812.836845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076813.156661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076813.860854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076814.180682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076814.884776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076815.204703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076815.908749:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076815.908760:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076815.908765:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076815.908776:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3505 00000100:00080000:2.0:1769076815.908788:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769076815.908791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076815.908797:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076815.908839:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923784448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076815.908876:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076815.908881:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076815.908884:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076815.908890:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3505 00000100:00080000:2.0:1769076815.908897:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076815.908905:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076815.908919:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076815.908924:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923784448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076815.908944:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923784576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076815.908989:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076815.908992:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076815.908994:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076815.908998:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076815.909004:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3505 00000100:00080000:2.0:1769076815.909011:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076815.909017:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076815.909024:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076815.909045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076815/real 1769076815] req@ffff969e0a46d880 x1855007923784448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076815.909047:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923784704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076815.909111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923784448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076815.909117:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076815.909123:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076815.909127:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076815.909141:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3505 00000100:00080000:1.0:1769076815.909147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076815.909149:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076815.909156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076815.909163:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076815.909166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923784448:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076815.909176:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076815.909197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076815.909201:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923784832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076815.909222:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076815.909227:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076815.909234:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076815.909235:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076815.909248:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3505 00000100:00100000:1.0:1769076815.909265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076815/real 1769076815] req@ffff969e0a46ea00 x1855007923784576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076815.909286:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076815.909320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923784576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076815.909327:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076815.909340:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076815.909346:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076815.909358:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076815.909370:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923784960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076815.909371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076815.909395:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076815.909457:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076815.909476:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076815/real 1769076815] req@ffff969e0a46ca80 x1855007923784704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076815.909495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923784704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076815.909507:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076815.909512:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076815.909517:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076815.909581:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076815.909620:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076815.909722:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076815/real 1769076815] req@ffff969e0a46df80 x1855007923784832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076815.909741:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923784832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076815.909754:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076815.909759:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076815.909780:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076815.909791:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076815.909823:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076815.909841:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076815/real 1769076815] req@ffff969e0a46ed80 x1855007923784960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076815.909860:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923784960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076815.909874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076815.909880:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076815.909885:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923784960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076816.228684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076816.932819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076817.252727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076817.956813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076818.276687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076818.980895:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076819.300754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076820.004773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076820.324764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076821.028877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076821.028906:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076821.028920:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076821.028925:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076821.028939:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3510 00000100:00080000:2.0:1769076821.028953:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076821.028973:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076821.029022:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923785088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076821.029132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923785088:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769076821.029211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076821.029260:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076821.029266:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076821.029269:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076821.029274:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3510 00000100:00080000:2.0:1769076821.029280:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076821.029296:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076821.029308:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076821.029350:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923785216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076821.029436:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076821/real 1769076821] req@ffff969e0a46e300 x1855007923785088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076821.029471:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923785088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076821.029484:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076821.029489:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076821.029495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923785088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076821.029525:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076821.029587:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076821.029666:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076821/real 1769076821] req@ffff969e0a46ce00 x1855007923785216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076821.029693:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923785216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076821.029703:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076821.029711:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076821.029711:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076821.029727:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076821.029729:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076821.029736:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3510 00000100:00100000:1.0:1769076821.029737:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785216:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076821.029745:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076821.029752:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076821.029764:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076821.029794:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923785344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076821.029828:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076821.029845:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076821.029849:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076821.029858:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3510 00000100:00080000:2.0:1769076821.029865:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076821.029872:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076821.029882:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076821.029895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076821.029913:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923785472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076821.029933:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076821.029939:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076821.029943:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076821.029946:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076821.029961:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3510 00000100:00100000:1.0:1769076821.029968:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076821/real 1769076821] req@ffff969e0a46ce00 x1855007923785344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076821.029968:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076821.029976:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076821.029986:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076821.029993:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923785344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076821.030007:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923785600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076821.030042:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076821.030051:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076821.030058:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076821.030086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076821.030128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076821.030144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076821/real 1769076821] req@ffff969e0a46e300 x1855007923785472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076821.030160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923785472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076821.030171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076821.030175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076821.030179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076821.030186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076821.030206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076821.030217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076821/real 1769076821] req@ffff969e0a46ed80 x1855007923785600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076821.030230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923785600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076821.030240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076821.030244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076821.030258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076821.348681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076822.053037:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076822.372697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076823.076860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076823.396755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076824.100911:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076824.420772:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076825.124775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076825.444693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076826.148780:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769076826.148780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076826.148799:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076826.148808:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076826.148844:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3515 00000100:00080000:0.0:1769076826.148865:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076826.148875:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076826.148935:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923785728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076826.148997:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076826.149003:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076826.149005:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076826.149012:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3515 00000100:00080000:0.0:1769076826.149019:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076826.149025:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076826.149039:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076826.149057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923785728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076826.149060:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923785856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076826.149094:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076826.149099:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076826.149103:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076826.149110:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3515 00000100:00080000:0.0:1769076826.149115:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076826.149121:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076826.149156:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076826.149163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076826.149178:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923785984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076826.149199:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076826.149204:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076826.149208:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076826.149214:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076826/real 1769076826] req@ffff969e06a61880 x1855007923785728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076826.149215:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3515 00000100:00080000:0.0:1769076826.149229:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076826.149257:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076826.149272:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076826.149288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923785728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076826.149289:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923786112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076826.149352:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076826.149355:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076826.149367:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076826.149368:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769076826.149375:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076826.149384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923785728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076826.149386:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3515 00000100:00080000:0.0:1769076826.149404:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076826.149416:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076826.149424:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076826.149442:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769076826.149443:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923786240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076826.149525:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076826.149557:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076826/real 1769076826] req@ffff969e06a63100 x1855007923785856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076826.149571:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923785856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076826.149580:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076826.149583:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076826.149586:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076826.149619:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076826.149637:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076826.149665:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076826/real 1769076826] req@ffff969e06a61f80 x1855007923785984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076826.149676:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923785984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076826.149684:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076826.149687:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076826.149690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923785984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076826.149695:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076826.149712:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076826.149720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076826/real 1769076826] req@ffff969e06a62680 x1855007923786112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076826.149730:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923786112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076826.149737:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076826.149739:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076826.149742:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076826.149758:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076826.149772:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076826.149779:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076826/real 1769076826] req@ffff969e06a61500 x1855007923786240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076826.149789:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923786240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076826.149802:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076826.149806:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076826.149809:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076826.468722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076827.172812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076827.492734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076828.196852:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076828.516730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076829.220817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076829.540692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076830.244762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076830.564659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076831.268772:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076831.268787:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076831.268793:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076831.268823:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3520 00010000:00010000:2.0:1769076831.268828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076831.268838:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076831.268852:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076831.268912:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923786368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076831.268969:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076831.268977:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076831.268982:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076831.268992:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3520 00000100:00080000:3.0:1769076831.269001:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076831.269040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923786368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076831.269053:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076831.269104:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076831.269121:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923786496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076831.269130:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076831.269133:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076831.269136:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076831.269139:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3520 00000100:00080000:3.0:1769076831.269143:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076831.269147:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076831.269151:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076831.269190:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923786624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076831.269207:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786496:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076831.269211:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076831.269236:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076831.269239:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076831.269246:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3520 00000100:00100000:1.0:1769076831.269253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786624:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769076831.269254:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076831.269271:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076831.269278:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076831.269310:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923786752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076831.269329:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076831.269338:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076831.269342:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076831.269349:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3520 00000100:00080000:3.0:1769076831.269356:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076831.269360:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076831.269366:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786752:192.168.204.143@tcp:8:kworker.0 00000008:00100000:3.0:1769076831.269367:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076831.269405:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923786880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076831.269430:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076831.588676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076832.292866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076832.612658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076833.316770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076833.636731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076834.340790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769076834.341018:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769076834.341112:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769076834.341223:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076834.341283:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076834.341300:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076834.341314:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769076834.341357:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076834.341456:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076831/real 1769076834] req@ffff969e0a7fdf80 x1855007923786880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076834.341499:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923786880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076834.341519:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076834.341528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076834.341535:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076834.341548:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076831/real 1769076834] req@ffff969e0a7fc380 x1855007923786752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076834.341568:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923786752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076834.341637:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076834.341644:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076834.341650:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076834.341662:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076831/real 1769076834] req@ffff969e0a7fe300 x1855007923786624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076834.341686:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923786624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076834.341698:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076834.341703:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076834.341709:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076834.341720:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076831/real 1769076834] req@ffff969e0a7fca80 x1855007923786496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076834.341736:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923786496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076834.341749:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076834.341754:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076834.341758:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923786496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076834.341769:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076831/real 1769076834] req@ffff969e0a7fd880 x1855007923786368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076834.341786:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923786368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076834.341809:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076834.341814:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076834.341819:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923786368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076834.660724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076835.364787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076835.684724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076836.388844:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076836.388860:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076836.388867:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076836.388880:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3525 00000100:00080000:0.0:1769076836.388894:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769076836.388900:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076836.388905:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076836.388952:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923787008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076836.388990:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076836.388995:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076836.388997:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076836.389003:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3525 00000100:00080000:0.0:1769076836.389007:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076836.389029:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076836.389042:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076836.389058:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923787136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076836.389077:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076836.389083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923787008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076836.389086:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076836.389116:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076836.389127:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3525 00000100:00080000:0.0:1769076836.389138:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076836.389146:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076836.389158:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076836.389191:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923787264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076836.389193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076836.389219:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076836.389227:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076836.389231:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076836.389240:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3525 00000100:00080000:0.0:1769076836.389248:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076836.389269:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076836.389271:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076836/real 1769076836] req@ffff969e06a61180 x1855007923787008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769076836.389280:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076836.389332:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923787392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076836.389377:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076836.389383:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076836.389388:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076836.389397:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923787008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076836.389408:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3525 00000100:00080000:0.0:1769076836.389425:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076836.389434:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076836.389438:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076836.389451:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:0.0:1769076836.389454:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076836.389464:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923787008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076836.389504:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923787520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076836.389533:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076836.389625:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076836.389654:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076836/real 1769076836] req@ffff969e06a60380 x1855007923787136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076836.389678:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923787136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076836.389695:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076836.389703:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076836.389708:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076836.389737:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076836.389781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076836.389803:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076836/real 1769076836] req@ffff969e06a60a80 x1855007923787264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076836.389828:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923787264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076836.389843:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076836.389850:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076836.389857:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076836.389868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076836.389907:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076836.389929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076836/real 1769076836] req@ffff969e06a60700 x1855007923787392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076836.389952:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923787392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076836.389997:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076836.390006:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076836.390013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076836.390046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076836.390090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076836.390114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076836/real 1769076836] req@ffff969e06a63800 x1855007923787520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076836.390155:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923787520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076836.390172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076836.390178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076836.390184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076836.708754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076837.412833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076837.732685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076838.436789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076838.756698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076839.460711:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076839.780642:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076840.484886:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076840.804770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076841.508716:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076841.508729:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076841.508735:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769076841.508747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076841.508747:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3530 00000100:00080000:3.0:1769076841.508761:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076841.508769:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076841.508817:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923787648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076841.508846:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076841.508851:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076841.508853:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076841.508874:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3530 00000100:00080000:3.0:1769076841.508879:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076841.508884:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076841.508896:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076841.508914:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923787776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076841.508923:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923787648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076841.508950:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076841.508955:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076841.508957:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076841.508962:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3530 00000100:00080000:3.0:1769076841.508968:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076841.508973:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076841.508980:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076841.508995:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923787904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076841.509007:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076841.509013:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076841.509019:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076841.509022:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076841.509028:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3530 00000100:00080000:3.0:1769076841.509033:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076841.509038:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076841.509045:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076841.509052:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076841/real 1769076841] req@ffff969e0a7fd180 x1855007923787648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076841.509079:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923788032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076841.509103:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076841.509106:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076841.509109:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076841.509112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923787648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076841.509117:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3530 00000100:00080000:1.0:1769076841.509123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076841.509130:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076841.509131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076841.509137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923787648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076841.509142:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076841.509149:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076841.509161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076841.509165:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923788160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076841.509186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076841.509215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076841/real 1769076841] req@ffff969e0a7ffb80 x1855007923787776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076841.509226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923787776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076841.509243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076841.509245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076841.509248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076841.509253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076841.509269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076841.509277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076841/real 1769076841] req@ffff969e0a7fdc00 x1855007923787904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076841.509284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923787904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076841.509290:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076841.509292:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076841.509294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923787904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076841.509298:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076841.509308:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076841.509314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076841/real 1769076841] req@ffff969e0a7ff800 x1855007923788032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076841.509320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923788032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076841.509326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076841.509328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076841.509335:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076841.509346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076841.509360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076841.509366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076841/real 1769076841] req@ffff969e0a7fc000 x1855007923788160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076841.509374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923788160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076841.509379:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076841.509381:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076841.509383:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076841.828667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076842.532774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076842.852730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076843.556801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076843.876716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076844.580703:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076844.900762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076845.604750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076845.924715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076846.628672:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076846.628682:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076846.628686:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076846.628695:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3535 00000100:00080000:3.0:1769076846.628705:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076846.628712:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076846.628749:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923788288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076846.628751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076846.628780:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076846.628783:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076846.628785:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076846.628788:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3535 00000100:00080000:3.0:1769076846.628791:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076846.628794:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076846.628803:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076846.628814:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923788416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076846.628823:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076846.628842:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076846.628846:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076846.628848:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923788288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076846.628851:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3535 00000100:00080000:3.0:1769076846.628858:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076846.628863:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076846.628868:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076846.628883:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923788544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076846.628896:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076846.628901:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076846.628903:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076846.628907:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3535 00000100:00080000:3.0:1769076846.628912:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076846.628917:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076846.628924:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076846.628937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076846.628943:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923788672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076846.628957:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076846.628960:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076846.628962:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076846.628967:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3535 00000100:00080000:3.0:1769076846.628971:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076846.628975:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076846/real 1769076846] req@ffff969e0a7fd500 x1855007923788288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076846.628979:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076846.628995:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076846.628996:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923788288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076846.629009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076846.629013:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923788800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076846.629017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076846.629027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923788288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076846.629040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076846.629060:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076846.629079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076846/real 1769076846] req@ffff969e0a7fc000 x1855007923788416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076846.629090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923788416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076846.629097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076846.629099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076846.629102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076846.629157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076846.629184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076846.629199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076846/real 1769076846] req@ffff969e0a7ff800 x1855007923788544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076846.629214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923788544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076846.629222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076846.629224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076846.629227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076846.629232:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076846.629250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076846.629258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076846/real 1769076846] req@ffff969e0a7fdc00 x1855007923788672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076846.629267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923788672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076846.629279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076846.629282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076846.629285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076846.629290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076846.629304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076846.629312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076846/real 1769076846] req@ffff969e0a7ffb80 x1855007923788800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076846.629320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923788800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076846.629327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076846.629329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076846.629331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923788800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076846.948717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076847.652854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076847.972718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076848.676770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076848.996703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076849.700795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076850.020712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076850.724757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076851.044708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076851.748715:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076851.748726:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076851.748731:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076851.748741:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3540 00000100:00080000:3.0:1769076851.748752:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076851.748760:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076851.748803:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923788928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076851.748824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076851.748839:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076851.748846:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076851.748849:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076851.748856:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3540 00000100:00080000:3.0:1769076851.748864:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076851.748869:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076851.748879:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076851.748884:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923788928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076851.748898:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923789056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076851.748911:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076851.748914:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076851.748916:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076851.748920:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3540 00000100:00080000:3.0:1769076851.748943:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076851.748947:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076851.748953:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076851.748954:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076851.748971:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923789184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076851.748984:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076851.748987:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076851.748989:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076851.748995:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076851/real 1769076851] req@ffff969e0a7fd180 x1855007923788928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076851.748997:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3540 00000100:00080000:3.0:1769076851.749006:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076851.749015:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076851.749021:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076851.749043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923788928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076851.749056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769076851.749059:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923789312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076851.749081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076851.749092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923788928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076851.749096:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076851.749099:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076851.749104:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076851.749113:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3540 00000100:00080000:3.0:1769076851.749117:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076851.749120:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076851.749125:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076851.749134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769076851.749139:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923789440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076851.749167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076851.749183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076851/real 1769076851] req@ffff969e0a7fd880 x1855007923789056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076851.749196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923789056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076851.749204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076851.749208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076851.749211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076851.749216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076851.749229:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076851.749236:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076851/real 1769076851] req@ffff969e0a7fca80 x1855007923789184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076851.749245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923789184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076851.749251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076851.749253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076851.749256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076851.749260:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076851.749272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076851.749279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076851/real 1769076851] req@ffff969e0a7fe300 x1855007923789312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076851.749296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923789312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076851.749308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076851.749312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076851.749316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076851.749337:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076851.749365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076851.749379:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076851/real 1769076851] req@ffff969e0a7fc380 x1855007923789440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076851.749395:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923789440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076851.749407:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076851.749412:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076851.749416:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076852.068670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076852.772877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076853.092783:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076853.796720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076854.116690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076854.820818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076855.140671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076855.844875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076856.164777:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076856.868742:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076856.868751:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076856.868755:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076856.868764:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3546 00000100:00080000:2.0:1769076856.868774:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076856.868781:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076856.868821:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923789568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076856.868825:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076856.868846:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076856.868850:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076856.868852:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076856.868856:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3546 00000100:00080000:2.0:1769076856.868859:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076856.868863:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076856.868872:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076856.868884:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923789696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076856.868890:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923789568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076856.868894:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076856.868902:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076856.868904:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076856.868907:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3546 00000100:00080000:2.0:1769076856.868909:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076856.868911:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076856.868914:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076856.868971:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923789824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076856.868980:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076856.868983:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076856.869000:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076856.869003:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3546 00000100:00080000:2.0:1769076856.869007:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076856.869012:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076856.869015:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076856.869022:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076856.869033:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923789952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076856.869042:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076856.869044:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076856.869046:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076856.869049:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3546 00000100:00080000:2.0:1769076856.869052:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076856.869056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076856/real 1769076856] req@ffff969e0a46df80 x1855007923789568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076856.869057:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076856.869063:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076856.869076:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923790080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076856.869089:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923789568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076856.869119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076856.869124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076856.869128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923789568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076856.869154:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076856.869180:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076856.869193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076856/real 1769076856] req@ffff969e0a46ca80 x1855007923789696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076856.869206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923789696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076856.869215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076856.869235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076856.869238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076856.869245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076856.869259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076856.869268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076856/real 1769076856] req@ffff969e0a46ea00 x1855007923789824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076856.869279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923789824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076856.869287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076856.869291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076856.869294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076856.869306:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076856.869335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076856.869345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076856/real 1769076856] req@ffff969e0a46d880 x1855007923789952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076856.869357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923789952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076856.869366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076856.869369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076856.869372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923789952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076856.869378:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076856.869394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076856.869402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076856/real 1769076856] req@ffff969e0a46f480 x1855007923790080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076856.869414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923790080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076856.869422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076856.869426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076856.869429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076857.188679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076857.892883:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076858.212823:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076858.916882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076859.236716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076859.941738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076860.260707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076860.965145:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076861.284705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076861.988919:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076861.988956:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076861.988965:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076861.988982:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3551 00000100:00080000:2.0:1769076861.988999:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076861.989011:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076861.989071:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923790208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076861.989127:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076861.989136:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076861.989141:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076861.989151:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3551 00000100:00080000:2.0:1769076861.989159:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076861.989174:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076861.989177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923790208:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076861.989192:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076861.989224:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923790336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076861.989248:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076861.989254:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076861.989257:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076861.989265:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3551 00000100:00080000:2.0:1769076861.989273:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076861.989282:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076861.989312:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076861.989337:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076861.989345:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923790464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076861.989370:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076861.989378:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076861.989382:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076861.989392:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3551 00000100:00080000:2.0:1769076861.989402:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076861.989410:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076861/real 1769076861] req@ffff969e0a46d500 x1855007923790208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076861.989417:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076861.989428:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076861.989477:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923790208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076861.989488:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923790592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076861.989501:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076861.989518:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076861.989524:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076861.989533:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923790208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076861.989537:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076861.989547:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076861.989560:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3551 00000100:00080000:2.0:1769076861.989568:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076861.989575:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076861.989587:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076861.989662:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076861.989680:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923790720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076861.989728:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076861.989776:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076861/real 1769076861] req@ffff969e0a46c380 x1855007923790336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076861.989801:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923790336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076861.989816:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076861.989822:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076861.989828:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076861.989839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076861.989901:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076861.989922:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076861/real 1769076861] req@ffff969e0a46f800 x1855007923790464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076861.989943:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923790464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076861.989982:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076861.989992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076861.990000:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076861.990014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076861.990061:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076861.990081:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076861/real 1769076861] req@ffff969e0a46f100 x1855007923790592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076861.990104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923790592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076861.990119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076861.990124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076861.990129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076861.990159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076861.990195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076861.990217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076861/real 1769076861] req@ffff969e0a46d500 x1855007923790720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076861.990240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923790720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076861.990254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076861.990259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076861.990278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790720:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769076861.990447:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076862.308663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076863.013584:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076863.332639:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076864.037366:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076864.356738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076865.060998:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076865.380682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076866.084852:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076866.404691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076867.108848:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076867.108868:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076867.108875:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076867.108892:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3556 00000100:00080000:3.0:1769076867.108912:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076867.108926:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076867.108959:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076867.108995:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923790848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076867.109048:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076867.109053:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076867.109056:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076867.109062:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3556 00000100:00080000:3.0:1769076867.109067:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076867.109073:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076867.109087:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076867.109107:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923790976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076867.109121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923790848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076867.109127:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076867.109142:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076867.109145:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076867.109152:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3556 00000100:00080000:3.0:1769076867.109156:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076867.109160:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076867.109194:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076867.109214:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923791104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076867.109228:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076867.109233:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076867.109235:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076867.109240:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3556 00000100:00080000:3.0:1769076867.109258:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076867.109263:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076867.109271:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076867.109281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076867.109292:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923791232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076867.109306:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076867.109312:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076867.109315:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076867.109321:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3556 00000100:00080000:3.0:1769076867.109328:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076867.109332:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076867.109338:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076867.109353:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923791360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076867.109369:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076867/real 1769076867] req@ffff969e0a7fdf80 x1855007923790848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076867.109415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923790848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076867.109432:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076867.109439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076867.109460:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923790848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076867.109500:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076867.109536:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076867.109554:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076867/real 1769076867] req@ffff969e0a7fea00 x1855007923790976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076867.109570:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923790976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076867.109582:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076867.109586:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076867.109626:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923790976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076867.109634:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076867.109659:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076867.109690:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076867/real 1769076867] req@ffff969e0a7fc700 x1855007923791104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076867.109706:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923791104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076867.109717:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076867.109721:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076867.109725:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076867.109733:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076867.109757:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076867.109801:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076867/real 1769076867] req@ffff969e0a7fce00 x1855007923791232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076867.109820:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923791232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076867.109832:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076867.109836:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076867.109840:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076867.109849:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076867.109876:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076867.109893:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076867/real 1769076867] req@ffff969e0a7fe680 x1855007923791360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076867.109918:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923791360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076867.109949:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076867.109957:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076867.109964:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076867.428756:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076868.132949:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076868.452712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076869.156896:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076869.476683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076870.180873:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076870.500678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076871.204888:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076871.524766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076872.228745:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076872.228754:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076872.228759:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076872.228768:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3561 00010000:00010000:0.0:1769076872.228774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076872.228779:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076872.228786:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076872.228820:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923791488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076872.228843:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076872.228845:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076872.228847:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076872.228850:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3561 00000100:00080000:2.0:1769076872.228852:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076872.228855:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076872.228862:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076872.228871:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923791616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076872.228880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923791488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076872.228881:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076872.228898:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076872.228900:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076872.228903:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3561 00000100:00080000:2.0:1769076872.228907:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076872.228910:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076872.228914:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076872.228924:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923791744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076872.228932:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076872.228935:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076872.228936:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076872.228938:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3561 00000100:00080000:2.0:1769076872.228940:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076872.228945:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769076872.228948:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769076872.228950:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076872.228959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923791872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076872.228966:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076872.228969:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076872.228970:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076872.228973:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3561 00000100:00080000:2.0:1769076872.228975:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076872.228978:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076872.228986:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076872.228989:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076872/real 1769076872] req@ffff969e0a46f480 x1855007923791488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076872.228999:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923792000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076872.229010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923791488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076872.229020:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076872.229024:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076872.229028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923791488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076872.229048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076872.229065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076872.229073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076872/real 1769076872] req@ffff969e0a46d880 x1855007923791616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076872.229080:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923791616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076872.229086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076872.229088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076872.229090:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076872.229101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076872.229116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076872.229122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076872/real 1769076872] req@ffff969e0a46ea00 x1855007923791744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076872.229129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923791744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076872.229135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076872.229136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076872.229138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076872.229142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076872.229152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076872.229157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076872/real 1769076872] req@ffff969e0a46ca80 x1855007923791872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076872.229164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923791872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076872.229169:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076872.229171:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076872.229173:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923791872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076872.229176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076872.229185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076872.229190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076872/real 1769076872] req@ffff969e0a46df80 x1855007923792000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076872.229203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923792000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076872.229209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076872.229211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076872.229213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076872.548663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076873.252877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076873.572721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076874.276804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076874.596766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076875.300878:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076875.620774:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076876.324858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076876.644723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076877.348816:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769076877.348827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076877.348843:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076877.348852:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076877.348868:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3566 00000100:00080000:0.0:1769076877.348923:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076877.348934:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076877.348992:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923792128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076877.349037:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076877.349043:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076877.349046:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076877.349052:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3566 00000100:00080000:0.0:1769076877.349056:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076877.349060:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076877.349074:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076877.349092:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923792256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076877.349107:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076877.349115:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076877.349119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923792128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076877.349120:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076877.349136:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3566 00000100:00080000:0.0:1769076877.349141:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076877.349145:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076877.349151:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076877.349168:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923792384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076877.349180:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076877.349193:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076877.349196:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076877.349202:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3566 00000100:00080000:0.0:1769076877.349208:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076877.349213:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076877.349220:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076877.349238:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923792512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076877.349250:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076877.349255:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076877.349257:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076877.349261:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3566 00000100:00080000:0.0:1769076877.349267:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076877.349271:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076877.349277:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076877.349293:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076877.349294:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923792640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076877.349373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076877/real 1769076877] req@ffff969e06a60000 x1855007923792128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076877.349433:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923792128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076877.349455:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076877.349464:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076877.349473:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923792128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076877.349518:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076877.349556:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076877.349575:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076877/real 1769076877] req@ffff969e06a62d80 x1855007923792256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076877.349664:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923792256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076877.349696:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076877.349701:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076877.349706:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076877.349716:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076877.349749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076877.349765:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076877/real 1769076877] req@ffff969e06a63800 x1855007923792384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076877.349780:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923792384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076877.349790:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076877.349795:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076877.349799:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076877.349806:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076877.349826:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076877.349837:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076877/real 1769076877] req@ffff969e06a60700 x1855007923792512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076877.349850:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923792512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076877.349862:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076877.349865:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076877.349879:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076877.349887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076877.349937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076877.349960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076877/real 1769076877] req@ffff969e06a60a80 x1855007923792640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076877.349986:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923792640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076877.350005:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076877.350011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076877.350019:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076877.668728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076878.372846:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076878.692700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076879.396859:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076879.716684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076880.420841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076880.740689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076881.444764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076881.764722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076882.468817:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076882.468832:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1769076882.468836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076882.468840:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076882.468855:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3571 00000100:00080000:0.0:1769076882.468870:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076882.468880:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076882.468934:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923792768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076882.469011:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076882.469016:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076882.469019:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076882.469025:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3571 00000100:00080000:0.0:1769076882.469033:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076882.469041:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076882.469056:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076882.469080:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923792896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076882.469089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923792768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076882.469111:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076882.469117:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076882.469119:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076882.469136:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3571 00000100:00080000:0.0:1769076882.469141:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076882.469146:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076882.469152:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076882.469169:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923793024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076882.469181:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076882.469185:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076882.469188:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076882.469192:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3571 00000100:00080000:0.0:1769076882.469206:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076882.469213:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076882.469222:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076882.469236:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923793152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076882.469250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076882.469266:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076882.469272:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076882.469276:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076882.469281:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3571 00000100:00080000:0.0:1769076882.469287:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076882.469292:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076882.469297:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076882.469311:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923793280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076882.469333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076882/real 1769076882] req@ffff969e06a60380 x1855007923792768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076882.469383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923792768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076882.469401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076882.469408:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076882.469432:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923792768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076882.469466:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076882.469496:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076882.469540:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076882/real 1769076882] req@ffff969e06a61180 x1855007923792896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076882.469555:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923792896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076882.469580:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076882.469587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076882.469594:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923792896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076882.469684:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076882.469715:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076882.469727:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076882/real 1769076882] req@ffff969e06a61500 x1855007923793024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076882.469738:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923793024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076882.469746:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076882.469750:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076882.469753:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076882.469759:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076882.469776:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076882.469794:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076882/real 1769076882] req@ffff969e06a62680 x1855007923793152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076882.469805:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923793152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076882.469814:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076882.469817:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076882.469820:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076882.469826:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076882.469843:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076882.469851:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076882/real 1769076882] req@ffff969e06a61f80 x1855007923793280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076882.469862:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923793280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076882.469874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076882.469879:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076882.469884:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076882.788730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076883.492806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076883.812737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076884.516891:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076884.836754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076885.540831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076885.860752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076886.564870:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076886.884803:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076887.588722:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076887.588734:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1769076887.588757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076887.588759:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076887.588770:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3576 00000100:00080000:3.0:1769076887.588780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076887.588786:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076887.588816:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923793408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076887.588841:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076887.588844:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076887.588845:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076887.588849:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3576 00000100:00080000:3.0:1769076887.588852:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076887.588856:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076887.588864:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076887.588870:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923793408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076887.588877:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923793536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076887.588886:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076887.588889:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076887.588890:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076887.588893:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3576 00000100:00080000:3.0:1769076887.588895:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076887.588897:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076887.588901:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076887.588910:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923793664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076887.588927:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076887.588930:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076887.588931:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076887.588934:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3576 00000100:00080000:3.0:1769076887.588940:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076887.588942:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076887.588946:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769076887.588953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769076887.588958:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923793792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076887.588966:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076887.588969:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076887.588970:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076887.588973:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3576 00000100:00080000:3.0:1769076887.588978:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076887.588980:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076887.588984:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076887.588991:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923793920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076887.588996:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076887/real 1769076887] req@ffff969e0a7fed80 x1855007923793408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076887.589026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923793408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076887.589034:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076887.589038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076887.589041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923793408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076887.589060:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076887.589079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076887.589088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076887/real 1769076887] req@ffff969e0a7fe680 x1855007923793536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076887.589097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923793536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076887.589103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076887.589105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076887.589108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076887.589112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769076887.589124:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076887.589139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076887/real 1769076887] req@ffff969e0a7fce00 x1855007923793664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076887.589148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923793664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076887.589157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076887.589160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076887.589164:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076887.589171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076887.589193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076887.589205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076887/real 1769076887] req@ffff969e0a7fc700 x1855007923793792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076887.589219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923793792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076887.589229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076887.589233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076887.589236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076887.589241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769076887.589256:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076887.589264:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076887/real 1769076887] req@ffff969e0a7fea00 x1855007923793920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076887.589272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923793920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076887.589284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076887.589287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769076887.589289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923793920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076887.908683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076888.612820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076888.932685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076889.636793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076889.956700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076890.660850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076890.980687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076891.684739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076892.004718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076892.708789:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076892.708808:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076892.708816:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076892.708833:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3581 00000100:00080000:2.0:1769076892.708863:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076892.708876:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076892.708904:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076892.708949:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923794048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076892.709000:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076892.709008:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076892.709011:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076892.709018:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3581 00000100:00080000:2.0:1769076892.709026:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076892.709062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923794048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076892.709063:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076892.709088:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076892.709126:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923794176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076892.709160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076892.709181:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076892.709187:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076892.709201:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076892.709210:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3581 00000100:00080000:2.0:1769076892.709219:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076892.709241:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076892.709243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076892/real 1769076892] req@ffff969e0a46ed80 x1855007923794048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076892.709251:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076892.709279:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923794304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076892.709313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923794048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076892.709320:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076892.709333:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076892.709361:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076892.709367:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3581 00000100:00080000:1.0:1769076892.709377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076892.709378:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076892.709391:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076892.709396:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076892.709401:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923794048:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769076892.709410:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076892.709441:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923794432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076892.709446:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794176:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076892.709472:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076892.709488:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076892.709491:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076892.709495:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076892.709502:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3581 00000100:00080000:2.0:1769076892.709511:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076892.709517:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076892.709524:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076892.709528:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076892/real 1769076892] req@ffff969e0a46e300 x1855007923794176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076892.709543:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923794560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076892.709549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923794176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076892.709588:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076892.709632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076892.709658:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076892.709674:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076892.709715:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076892.709732:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076892/real 1769076892] req@ffff969e0a46ce00 x1855007923794304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076892.709749:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923794304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076892.709759:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076892.709764:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076892.709768:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076892.709789:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076892.709811:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076892.709823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076892/real 1769076892] req@ffff969e0a46df80 x1855007923794432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076892.709837:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923794432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076892.709868:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076892.709872:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076892.709877:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076892.709884:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076892.709908:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076892.709926:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076892/real 1769076892] req@ffff969e0a46ca80 x1855007923794560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076892.709940:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923794560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076892.709950:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076892.709954:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076892.709957:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076893.028742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076893.732868:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076894.052709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076894.756779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076895.076721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076895.780784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076896.100729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076896.804800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076897.124749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076897.828696:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076897.828709:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076897.828713:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076897.828724:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3587 00000100:00080000:2.0:1769076897.828734:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076897.828742:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076897.828756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076897.828781:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923794688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076897.828811:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076897.828815:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076897.828829:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076897.828835:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3587 00000100:00100000:1.0:1769076897.828852:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923794688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076897.828856:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076897.828864:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076897.828874:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076897.828902:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923794816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076897.828915:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076897.828919:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076897.828921:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076897.828925:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3587 00000100:00080000:2.0:1769076897.828930:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076897.828934:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076897.828940:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076897.828952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076897.828956:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923794944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076897.828981:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076897.828985:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076897.828988:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076897.828994:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794944:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769076897.828996:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3587 00000100:00080000:2.0:1769076897.829005:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076897.829010:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076897.829017:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076897.829043:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923795072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076897.829085:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076897.829100:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076897.829103:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076897.829105:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795072:192.168.204.143@tcp:8:kworker.0 00000100:00080000:2.0:1769076897.829112:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3587 00000100:00080000:2.0:1769076897.829118:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076897.829124:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076897.829132:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076897.829148:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923795200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076897.829202:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076898.148670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076898.852817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076899.172721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076899.876948:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076900.196694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076900.900817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1769076900.900945:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:3.0:1769076900.901006:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769076900.901033:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076900.901076:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076900.901108:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076900.901116:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076897/real 1769076900] req@ffff969e0a46ea00 x1855007923794688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1769076900.901117:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076900.901129:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076900.901159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923794688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076900.901170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076900.901176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076900.901186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923794688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076900.901237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076897/real 1769076900] req@ffff969e0a46f100 x1855007923795200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076900.901249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923795200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076900.901257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076900.901260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076900.901262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076900.901267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076897/real 1769076900] req@ffff969e0a46d500 x1855007923795072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076900.901276:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923795072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076900.901281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076900.901283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076900.901285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076900.901290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076897/real 1769076900] req@ffff969e0a46f480 x1855007923794944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076900.901309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923794944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076900.901316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076900.901319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076900.901321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076900.901325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076897/real 1769076900] req@ffff969e0a46d880 x1855007923794816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076900.901332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923794816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076900.901338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076900.901341:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076900.901343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923794816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076901.220771:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076901.924794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076902.244723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076902.948724:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076902.948736:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076902.948740:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076902.948751:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3592 00000100:00080000:3.0:1769076902.948763:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076902.948769:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076902.948800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076902.948807:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923795328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076902.948836:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076902.948839:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076902.948841:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076902.948845:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3592 00000100:00080000:3.0:1769076902.948848:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076902.948851:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076902.948860:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076902.948871:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923795456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076902.948881:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076902.948886:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076902.948889:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076902.948895:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3592 00000100:00100000:1.0:1769076902.948905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923795328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076902.948934:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076902.948944:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076902.948953:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076902.948984:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923795584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076902.948996:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076902.949007:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076902.949013:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076902.949016:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076902.949023:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3592 00000100:00080000:3.0:1769076902.949030:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076902.949035:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076902.949043:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076902.949047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076902/real 1769076902] req@ffff969e0a7fdf80 x1855007923795328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076902.949062:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923795712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076902.949079:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076902.949081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923795328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076902.949090:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076902.949102:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076902.949104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076902.949112:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3592 00000100:00080000:1.0:1769076902.949117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076902.949121:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076902.949129:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076902.949140:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076902.949141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923795328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076902.949163:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923795840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076902.949186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076902.949236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076902.949259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076902/real 1769076902] req@ffff969e0a7fc380 x1855007923795456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076902.949279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923795456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076902.949291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076902.949296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076902.949301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076902.949333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076902.949361:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076902.949377:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076902/real 1769076902] req@ffff969e0a7fe300 x1855007923795584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076902.949394:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923795584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076902.949406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076902.949411:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076902.949416:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076902.949424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076902.949449:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076902.949475:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076902/real 1769076902] req@ffff969e0a7fca80 x1855007923795712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076902.949494:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923795712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076902.949506:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076902.949511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076902.949516:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076902.949524:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076902.949552:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076902.949567:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076902/real 1769076902] req@ffff969e0a7fd880 x1855007923795840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076902.949583:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923795840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076902.949623:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076902.949628:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076902.949633:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923795840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076903.268685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076903.972765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076904.292704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076904.996866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076905.316692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076906.020825:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076906.340880:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076907.044789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076907.364712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076908.068736:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076908.068750:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076908.068756:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769076908.068769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076908.068769:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3597 00000100:00080000:3.0:1769076908.068790:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076908.068798:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076908.068845:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923795968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076908.068874:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076908.068878:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076908.068880:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076908.068886:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3597 00000100:00080000:3.0:1769076908.068890:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076908.068894:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076908.068904:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076908.068919:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923796096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076908.068920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923795968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076908.068951:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076908.068955:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076908.068956:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076908.068960:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3597 00000100:00080000:3.0:1769076908.068965:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076908.068970:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076908.069007:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076908.069020:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923796224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076908.069032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076908.069059:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076908.069062:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076908.069064:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076908.069068:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3597 00000100:00080000:3.0:1769076908.069073:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076908.069077:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076908.069097:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076908.069103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076908/real 1769076908] req@ffff969e0a7fd180 x1855007923795968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076908.069112:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923796352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076908.069144:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076908.069147:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769076908.069150:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076908.069156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923795968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076908.069157:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3597 00000100:00080000:3.0:1769076908.069162:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076908.069165:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076908.069172:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076908.069189:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923796480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076908.069198:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076908.069208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076908.069215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923795968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076908.069244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076908.069269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076908.069279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076908/real 1769076908] req@ffff969e0a7ffb80 x1855007923796096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076908.069288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923796096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076908.069295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076908.069298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076908.069300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076908.069305:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076908.069316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076908.069322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076908/real 1769076908] req@ffff969e0a7fdc00 x1855007923796224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076908.069329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923796224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076908.069335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076908.069336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076908.069338:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076908.069349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076908.069363:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076908.069369:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076908/real 1769076908] req@ffff969e0a7ff800 x1855007923796352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076908.069379:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923796352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076908.069388:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076908.069392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076908.069395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076908.069401:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076908.069421:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076908.069432:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076908/real 1769076908] req@ffff969e0a7fc000 x1855007923796480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076908.069443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923796480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076908.069449:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076908.069451:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076908.069454:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076908.388676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076909.092761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076909.412706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076910.116823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076910.436725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076911.140824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076911.460678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076912.164831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076912.484672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076913.188757:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076913.188770:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076913.188776:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076913.188788:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3602 00000100:00080000:0.0:1769076913.188800:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769076913.188803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076913.188810:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076913.188856:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923796608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076913.188892:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076913.188913:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076913.188916:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076913.188923:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3602 00000100:00080000:0.0:1769076913.188930:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076913.188936:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076913.188947:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076913.188956:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923796608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076913.188968:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923796736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076913.188983:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076913.188987:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076913.189024:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076913.189030:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3602 00000100:00080000:0.0:1769076913.189053:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076913.189057:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076913.189064:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076913.189069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769076913.189087:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923796864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076913.189100:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076913.189103:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076913.189106:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076913.189112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076913/real 1769076913] req@ffff969e06a63100 x1855007923796608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076913.189115:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3602 00000100:00080000:0.0:1769076913.189119:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076913.189130:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076913.189137:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076913.189155:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923796992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076913.189166:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076913.189168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923796608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076913.189174:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076913.189179:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076913.189191:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3602 00000100:00080000:0.0:1769076913.189198:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076913.189199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076913.189204:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076913.189211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:0.0:1769076913.189212:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076913.189218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923796608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076913.189250:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923797120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076913.189266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076913.189322:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076913.189338:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076913/real 1769076913] req@ffff969e06a61880 x1855007923796736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076913.189352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923796736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076913.189359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076913.189362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076913.189364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076913.189369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076913.189385:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076913.189393:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076913/real 1769076913] req@ffff969e06a61f80 x1855007923796864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076913.189402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923796864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076913.189420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076913.189423:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076913.189425:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076913.189430:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076913.189446:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076913.189454:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076913/real 1769076913] req@ffff969e06a62680 x1855007923796992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076913.189462:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923796992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076913.189469:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076913.189471:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076913.189473:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923796992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076913.189485:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076913.189498:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076913.189512:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076913/real 1769076913] req@ffff969e06a61500 x1855007923797120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076913.189521:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923797120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076913.189527:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076913.189529:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076913.189537:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076913.508680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076914.212723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076914.532794:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076915.236760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076915.556665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076916.260805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076916.580690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076917.284831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076917.604671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076918.308735:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076918.308747:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1769076918.308750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076918.308752:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076918.308764:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3607 00000100:00080000:2.0:1769076918.308774:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076918.308780:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076918.308812:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923797248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076918.308836:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076918.308839:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076918.308840:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076918.308843:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3607 00000100:00080000:2.0:1769076918.308845:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076918.308848:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076918.308868:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076918.308876:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923797376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076918.308884:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923797248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076918.308887:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076918.308892:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076918.308894:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076918.308928:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3607 00000100:00080000:2.0:1769076918.308932:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076918.308935:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076918.308939:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076918.308959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923797504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076918.308967:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076918.308969:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076918.308971:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076918.308974:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3607 00000100:00080000:2.0:1769076918.308976:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076918.308978:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076918.308982:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076918.308987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076918.308993:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923797632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076918.309001:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076918.309003:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076918.309004:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076918.309007:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3607 00000100:00080000:2.0:1769076918.309010:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076918.309012:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076918.309016:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076918.309020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076918/real 1769076918] req@ffff969e0a46f800 x1855007923797248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076918.309024:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923797760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076918.309040:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923797248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076918.309049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076918.309054:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076918.309057:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923797248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076918.309079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076918.309104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076918.309118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076918/real 1769076918] req@ffff969e0a46c380 x1855007923797376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076918.309132:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923797376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076918.309138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076918.309141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076918.309143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076918.309147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076918.309173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076918.309188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076918/real 1769076918] req@ffff969e0a46d880 x1855007923797504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076918.309197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923797504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076918.309203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076918.309205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076918.309208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076918.309214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076918.309245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076918.309256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076918/real 1769076918] req@ffff969e0a46f480 x1855007923797632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076918.309270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923797632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076918.309279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076918.309281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076918.309283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076918.309288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076918.309303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076918.309312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076918/real 1769076918] req@ffff969e0a46d500 x1855007923797760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076918.309324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923797760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076918.309342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076918.309345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076918.309349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923797760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076918.628719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076919.332838:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076919.652665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076920.356738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076920.676711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076921.380810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076921.700707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076922.404832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076922.724708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076923.428766:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076923.428778:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076923.428784:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076923.428796:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3612 00000100:00080000:2.0:1769076923.428809:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076923.428817:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076923.428909:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076923.428920:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923797888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076923.428941:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076923.428943:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076923.428945:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076923.428948:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3612 00000100:00080000:2.0:1769076923.428951:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076923.428955:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076923.428963:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076923.428973:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923797888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076923.428974:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923798016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076923.428987:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076923.428989:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076923.429004:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076923.429008:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3612 00000100:00080000:2.0:1769076923.429011:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076923.429014:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076923.429028:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076923.429038:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923798144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076923.429045:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076923.429047:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076923.429048:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076923.429050:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3612 00000100:00080000:2.0:1769076923.429053:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076923.429057:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076923.429058:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076923.429064:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076923.429090:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923798272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076923.429102:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076923.429106:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076923.429109:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076923.429113:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3612 00000100:00080000:2.0:1769076923.429118:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076923.429123:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076923/real 1769076923] req@ffff969e0a46f100 x1855007923797888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076923.429126:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076923.429132:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076923.429152:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923798400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076923.429202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923797888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076923.429234:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076923.429249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076923.429252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923797888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076923.429276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076923.429342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076923.429366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076923/real 1769076923] req@ffff969e0a46ea00 x1855007923798016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076923.429389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923798016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076923.429400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076923.429434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076923.429439:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076923.429469:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076923.429513:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076923.429528:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076923/real 1769076923] req@ffff969e0a46ca80 x1855007923798144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076923.429536:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923798144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076923.429541:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076923.429543:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076923.429546:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076923.429549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076923.429582:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076923.429591:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076923/real 1769076923] req@ffff969e0a46df80 x1855007923798272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076923.429603:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923798272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076923.429613:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076923.429616:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076923.429619:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076923.429625:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076923.429684:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076923.429696:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076923/real 1769076923] req@ffff969e0a46ce00 x1855007923798400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076923.429728:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923798400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076923.429738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076923.429741:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076923.429745:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076923.748717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076924.452843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076924.772664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076925.477021:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076925.796686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076926.501144:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076926.820716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076927.526155:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076927.844741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076928.548712:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076928.548727:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076928.548734:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076928.548777:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3617 00000100:00080000:2.0:1769076928.548789:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076928.548809:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076928.548851:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923798528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076928.548877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076928.548890:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076928.548895:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076928.548897:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076928.548901:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3617 00000100:00080000:2.0:1769076928.548906:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076928.548911:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076928.548923:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076928.548936:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923798528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076928.548939:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923798656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076928.548973:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076928.548977:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076928.548979:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076928.548985:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3617 00000100:00080000:2.0:1769076928.548991:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076928.548995:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076928.549001:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076928.549027:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923798784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076928.549036:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076928.549040:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076928.549043:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076928.549047:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076928.549048:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3617 00000100:00080000:2.0:1769076928.549056:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076928.549060:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076928.549065:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076928.549076:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923798912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076928.549086:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769076928.549089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076928/real 1769076928] req@ffff969e0a46e300 x1855007923798528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076928.549090:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076928.549094:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076928.549108:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3617 00000100:00100000:1.0:1769076928.549115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923798528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076928.549121:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076928.549131:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076928.549135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076928.549138:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076928.549145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076928.549151:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923798528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076928.549157:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923799040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076928.549182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076928.549208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076928.549219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076928/real 1769076928] req@ffff969e0a46ed80 x1855007923798656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076928.549227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923798656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076928.549243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076928.549246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076928.549248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076928.549253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076928.549267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076928.549274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076928/real 1769076928] req@ffff969e0a46ce00 x1855007923798784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076928.549281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923798784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076928.549286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076928.549288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076928.549290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076928.549293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076928.549304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076928.549309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076928/real 1769076928] req@ffff969e0a46df80 x1855007923798912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076928.549316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923798912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076928.549321:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076928.549323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076928.549330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923798912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076928.549334:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076928.549347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076928.549353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076928/real 1769076928] req@ffff969e0a46ca80 x1855007923799040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076928.549361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923799040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076928.549366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076928.549368:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076928.549370:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076928.868721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076929.573255:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076929.892745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076930.597163:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076930.916734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076931.620910:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076931.940770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076932.644743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076932.964681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076933.668732:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076933.668756:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076933.668762:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076933.668773:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3622 00000100:00080000:2.0:1769076933.668800:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076933.668808:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076933.668843:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923799168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769076933.668864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076933.668888:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076933.668892:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076933.668893:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076933.668897:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3622 00000100:00080000:2.0:1769076933.668901:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076933.668905:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076933.668914:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076933.668945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923799168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076933.668967:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923799296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076933.668980:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076933.668984:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076933.668986:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076933.668991:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3622 00000100:00080000:2.0:1769076933.668996:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076933.669001:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076933.669007:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076933.669014:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076933.669023:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923799424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076933.669032:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076933.669036:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076933.669038:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076933.669041:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3622 00000100:00080000:2.0:1769076933.669045:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076933.669049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076933/real 1769076933] req@ffff969e0a46ea00 x1855007923799168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076933.669052:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076933.669060:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076933.669083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923799168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076933.669085:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923799552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076933.669106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076933.669107:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076933.669112:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076933.669115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769076933.669116:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076933.669120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923799168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076933.669131:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3622 00000100:00080000:2.0:1769076933.669137:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076933.669142:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076933.669147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799296:192.168.204.143@tcp:38:kworker.0 00000008:00100000:2.0:1769076933.669150:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076933.669169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076933.669170:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923799680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076933.669191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076933/real 1769076933] req@ffff969e0a46f100 x1855007923799296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076933.669230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923799296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076933.669237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076933.669240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076933.669243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076933.669248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076933.669264:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076933.669288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076933/real 1769076933] req@ffff969e0a46d500 x1855007923799424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076933.669296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923799424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076933.669301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076933.669303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076933.669305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076933.669309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076933.669320:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076933.669326:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076933/real 1769076933] req@ffff969e0a46f480 x1855007923799552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076933.669333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923799552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076933.669344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076933.669346:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076933.669348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076933.669361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076933.669374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076933.669390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076933/real 1769076933] req@ffff969e0a46d880 x1855007923799680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076933.669397:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923799680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076933.669402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076933.669404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076933.669406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076933.988717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076934.692805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076935.012670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076935.716891:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076936.036729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076936.740824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076937.060718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076937.764778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076938.084745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076938.788763:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076938.788774:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076938.788779:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076938.788790:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3627 00000100:00080000:0.0:1769076938.788819:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076938.788827:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769076938.788846:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076938.788881:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923799808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076938.788926:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076938.788930:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076938.788932:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076938.788936:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3627 00000100:00080000:0.0:1769076938.788940:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076938.788944:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076938.788954:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076938.788972:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923799936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076938.788973:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923799808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076938.788991:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076938.788995:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076938.788997:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076938.789002:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3627 00000100:00080000:0.0:1769076938.789008:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076938.789014:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076938.789022:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076938.789048:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923800064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076938.789060:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076938.789068:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076938.789074:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076938.789078:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076938.789085:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3627 00000100:00100000:1.0:1769076938.789101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076938/real 1769076938] req@ffff969e06a61180 x1855007923799808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076938.789102:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076938.789116:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076938.789127:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076938.789128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923799808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076938.789142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076938.789149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769076938.789166:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923800192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076938.789172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923799808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769076938.789185:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076938.789203:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076938.789206:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076938.789213:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3627 00000100:00080000:0.0:1769076938.789220:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076938.789226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799936:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769076938.789229:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076938.789239:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076938.789258:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923800320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076938.789263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076938.789281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076938/real 1769076938] req@ffff969e06a60380 x1855007923799936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076938.789296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923799936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076938.789305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076938.789310:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076938.789315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923799936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076938.789323:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076938.789351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076938.789367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076938/real 1769076938] req@ffff969e06a60a80 x1855007923800064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076938.789418:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923800064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076938.789430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076938.789434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076938.789437:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076938.789443:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076938.789482:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076938.789494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076938/real 1769076938] req@ffff969e06a60700 x1855007923800192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076938.789514:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923800192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076938.789522:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076938.789525:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076938.789528:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076938.789546:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076938.789636:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076938.789656:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076938/real 1769076938] req@ffff969e06a63800 x1855007923800320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076938.789674:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923800320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076938.789687:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076938.789692:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076938.789697:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076939.108707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076939.812755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076940.132702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076940.836803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076941.156757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076941.860854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076942.180650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076942.884910:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076943.204708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076943.908715:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076943.908726:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076943.908730:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076943.908740:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3633 00000100:00080000:2.0:1769076943.908762:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076943.908770:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076943.908772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076943.908810:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923800448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076943.908854:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076943.908857:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076943.908859:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076943.908863:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3633 00000100:00080000:2.0:1769076943.908866:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076943.908870:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076943.908880:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076943.908892:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923800576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076943.908912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923800448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076943.908917:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076943.908922:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076943.908924:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076943.908928:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3633 00000100:00080000:2.0:1769076943.908931:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076943.908935:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076943.908939:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076943.908950:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923800704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076943.908958:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076943.908961:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076943.908962:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076943.908966:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3633 00000100:00080000:2.0:1769076943.908976:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076943.908984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076943.908985:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076943.908994:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076943.909006:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923800832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076943.909015:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076943.909019:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076943.909021:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076943.909025:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076943/real 1769076943] req@ffff969e0a46c380 x1855007923800448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076943.909029:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3633 00000100:00080000:2.0:1769076943.909035:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076943.909043:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076943.909048:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076943.909056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923800448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076943.909060:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923800960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076943.909097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076943.909103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076943.909109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923800448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076943.909137:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076943.909169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076943.909195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076943/real 1769076943] req@ffff969e0a46f800 x1855007923800576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076943.909207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923800576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076943.909215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076943.909217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076943.909220:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076943.909226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076943.909243:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076943.909252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076943/real 1769076943] req@ffff969e0a46d880 x1855007923800704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076943.909261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923800704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076943.909268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076943.909270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076943.909272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076943.909285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076943.909302:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076943.909312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076943/real 1769076943] req@ffff969e0a46f480 x1855007923800832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076943.909327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923800832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076943.909339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076943.909344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076943.909348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076943.909356:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076943.909382:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076943.909404:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076943/real 1769076943] req@ffff969e0a46d500 x1855007923800960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076943.909415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923800960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076943.909423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076943.909426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076943.909428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923800960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076944.228710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076944.932799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076945.252798:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076945.956819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076946.276740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076946.980841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076947.300651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076948.004877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076948.324700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076949.028785:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076949.028818:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1769076949.028819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076949.028824:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076949.028849:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3638 00000100:00080000:0.0:1769076949.028862:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076949.028870:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076949.028979:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923801088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076949.029030:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076949.029034:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076949.029036:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076949.029040:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3638 00000100:00080000:0.0:1769076949.029045:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076949.029050:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076949.029060:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076949.029073:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923801088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076949.029075:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923801216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076949.029114:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076949.029118:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076949.029120:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076949.029125:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3638 00000100:00080000:0.0:1769076949.029129:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076949.029162:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076949.029168:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076949.029183:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923801344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076949.029197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076949.029200:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076949.029205:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076949.029225:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076949.029237:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3638 00000100:00080000:0.0:1769076949.029243:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076949.029248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076949/real 1769076949] req@ffff969e06a62d80 x1855007923801088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076949.029250:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076949.029255:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076949.029266:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923801472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076949.029344:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076949.029348:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076949.029351:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076949.029355:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3638 00000100:00080000:0.0:1769076949.029363:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076949.029363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923801088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076949.029376:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076949.029388:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076949.029411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769076949.029412:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923801600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076949.029441:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076949.029451:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923801088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076949.029482:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076949.029505:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076949.029515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076949/real 1769076949] req@ffff969e06a60000 x1855007923801216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076949.029526:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923801216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076949.029533:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076949.029535:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076949.029538:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076949.029543:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076949.029556:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076949.029591:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076949/real 1769076949] req@ffff969e06a63800 x1855007923801344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076949.029599:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923801344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076949.029614:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076949.029617:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076949.029639:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076949.029644:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076949.029662:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076949.029677:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076949/real 1769076949] req@ffff969e06a60700 x1855007923801472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076949.029685:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923801472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076949.029691:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076949.029694:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076949.029696:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076949.029700:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076949.029710:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076949.029716:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076949/real 1769076949] req@ffff969e06a60a80 x1855007923801600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076949.029723:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923801600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076949.029738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076949.029740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076949.029747:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076949.348753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076950.052893:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076950.372663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076951.076729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076951.396676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076952.100805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076952.420706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076953.124788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076953.444718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769076954.148715:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076954.148727:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076954.148752:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076954.148780:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3643 00000100:00080000:3.0:1769076954.148794:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769076954.148799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076954.148805:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769076954.148878:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923801728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076954.148927:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076954.148931:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076954.148935:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076954.148941:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3643 00000100:00080000:3.0:1769076954.148947:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076954.148953:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076954.148965:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076954.149008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923801728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769076954.149009:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923801856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076954.149024:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076954.149029:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076954.149032:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076954.149037:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3643 00000100:00080000:3.0:1769076954.149042:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076954.149047:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076954.149053:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076954.149066:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923801984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076954.149112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769076954.149128:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076954.149134:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769076954.149138:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076954.149146:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3643 00000100:00080000:3.0:1769076954.149153:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076954.149170:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076954.149176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076954/real 1769076954] req@ffff969e0a7fd500 x1855007923801728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769076954.149180:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076954.149200:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923802112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076954.149224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923801728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076954.149231:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076954.149235:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769076954.149236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076954.149237:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076954.149241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769076954.149241:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3643 00000100:00080000:3.0:1769076954.149245:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076954.149247:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923801728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769076954.149249:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769076954.149254:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769076954.149264:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923802240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076954.149271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076954.149296:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076954.149335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076954/real 1769076954] req@ffff969e0a7fc000 x1855007923801856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076954.149366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923801856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076954.149375:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076954.149377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076954.149381:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076954.149386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076954.149406:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076954.149425:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076954/real 1769076954] req@ffff969e0a7ff800 x1855007923801984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076954.149436:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923801984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076954.149443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076954.149446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076954.149449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923801984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076954.149454:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076954.149473:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076954.149486:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076954/real 1769076954] req@ffff969e0a7fdc00 x1855007923802112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076954.149502:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923802112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076954.149514:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076954.149519:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076954.149524:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076954.149547:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076954.149571:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076954.149581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076954/real 1769076954] req@ffff969e0a7ffb80 x1855007923802240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076954.149591:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923802240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076954.149628:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076954.149632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076954.149635:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076954.468762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076955.172910:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076955.492692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076956.196837:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076956.516747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076957.220871:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076957.540719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076958.244750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076958.564681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076959.268685:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076959.268694:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076959.268698:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076959.268706:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3648 00000100:00080000:0.0:1769076959.268715:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076959.268721:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769076959.268753:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923802368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769076959.268760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076959.268776:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076959.268794:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076959.268796:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076959.268800:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3648 00000100:00080000:0.0:1769076959.268803:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076959.268806:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076959.268812:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923802368:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769076959.268816:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076959.268827:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923802496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076959.268836:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076959.268838:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076959.268839:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076959.268842:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3648 00000100:00080000:0.0:1769076959.268845:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076959.268847:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076959.268850:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076959.268857:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923802624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076959.268863:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076959.268866:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076959.268867:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076959.268888:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076959.268890:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3648 00000100:00080000:0.0:1769076959.268897:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076959.268901:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076959.268913:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076959.268925:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923802752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076959.268935:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076959/real 1769076959] req@ffff969e06a60380 x1855007923802368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076959.268937:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076959.268948:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076959.268953:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076959.268963:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3648 00000100:00080000:0.0:1769076959.268968:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076959.268975:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923802368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076959.268976:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076959.268985:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076959.268999:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923802880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076959.269004:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076959.269019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076959.269027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923802368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076959.269059:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076959.269090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076959.269104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076959/real 1769076959] req@ffff969e06a61180 x1855007923802496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076959.269116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923802496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076959.269124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076959.269127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076959.269130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076959.269146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076959.269166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076959.269175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076959/real 1769076959] req@ffff969e06a61500 x1855007923802624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076959.269186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923802624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076959.269193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076959.269195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076959.269198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076959.269203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076959.269218:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076959.269283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076959/real 1769076959] req@ffff969e06a62680 x1855007923802752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076959.269291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923802752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076959.269297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076959.269300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076959.269302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076959.269306:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076959.269326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076959.269333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076959/real 1769076959] req@ffff969e06a61f80 x1855007923802880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076959.269347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923802880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076959.269353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076959.269355:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076959.269357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923802880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076959.588743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076960.292885:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076960.612695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076961.316901:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076961.636775:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076962.340835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076962.660677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076963.364814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076963.684650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076964.388729:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076964.388737:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076964.388742:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076964.388749:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3653 00000100:00080000:2.0:1769076964.388759:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076964.388765:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769076964.388779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076964.388799:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923803008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076964.388836:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076964.388838:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076964.388840:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076964.388843:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3653 00000100:00080000:2.0:1769076964.388846:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076964.388849:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076964.388856:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076964.388866:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923803136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076964.388874:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076964.388876:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076964.388877:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076964.388881:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3653 00000100:00080000:2.0:1769076964.388883:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076964.388886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923803008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076964.388888:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076964.388893:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076964.388901:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923803264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076964.388906:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076964.388909:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076964.388911:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076964.388913:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3653 00000100:00080000:2.0:1769076964.388922:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076964.388926:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076964.388930:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076964.388939:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923803392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076964.388946:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076964.388948:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076964.388949:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076964.388951:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3653 00000100:00080000:2.0:1769076964.388954:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076964.388956:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076964.388959:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076964.388965:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923803520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076964.389001:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076964.389040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076964.389067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076964.389091:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076964.708659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076965.412871:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076965.732751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076966.436903:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076966.756750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076967.460893:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1769076967.461054:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:3.0:1769076967.461181:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769076967.461213:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076967.461282:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076967.461332:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769076967.461346:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076967.461348:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076964/real 1769076967] req@ffff969e0a46f100 x1855007923803008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1769076967.461376:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769076967.461415:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923803008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076967.461433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076967.461458:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076967.461464:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923803008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076967.461504:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076964/real 1769076967] req@ffff969e0a46ea00 x1855007923803136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076967.461549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923803136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076967.461562:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076967.461567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076967.461572:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076967.461582:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076964/real 1769076967] req@ffff969e0a46ca80 x1855007923803264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076967.461628:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923803264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076967.461638:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076967.461643:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076967.461648:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076967.461659:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076964/real 1769076967] req@ffff969e0a46df80 x1855007923803392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076967.461680:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923803392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076967.461693:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076967.461783:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076967.461800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076967.461825:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076964/real 1769076967] req@ffff969e0a46ce00 x1855007923803520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076967.461895:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923803520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076967.461917:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076967.461924:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076967.461929:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076967.780689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076968.484722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076968.804706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076969.508798:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076969.508809:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076969.508814:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076969.508823:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3658 00000100:00080000:2.0:1769076969.508845:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076969.508852:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076969.508904:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923803648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769076969.508933:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076969.508934:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076969.508943:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076969.508946:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076969.508950:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3658 00000100:00080000:2.0:1769076969.508954:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076969.508958:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076969.508969:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076969.508981:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923803776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076969.508991:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076969.508994:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076969.508998:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076969.509002:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923803648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076969.509004:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3658 00000100:00080000:2.0:1769076969.509008:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076969.509017:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076969.509026:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076969.509034:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923803904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076969.509086:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076969.509090:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076969.509091:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076969.509102:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3658 00000100:00080000:2.0:1769076969.509106:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076969.509110:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076969.509115:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076969.509126:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923804032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076969.509132:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076969.509135:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076969.509136:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076969.509145:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3658 00000100:00080000:2.0:1769076969.509148:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076969.509150:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076969.509155:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769076969.509164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076969.509165:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923804160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076969.509230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076969/real 1769076969] req@ffff969e0a46ed80 x1855007923803648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076969.509261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923803648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076969.509277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076969.509283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076969.509290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923803648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076969.509331:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076969.509365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076969.509384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076969/real 1769076969] req@ffff969e0a46e300 x1855007923803776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076969.509400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923803776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076969.509439:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076969.509447:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076969.509452:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076969.509464:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076969.509495:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076969.509511:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076969/real 1769076969] req@ffff969e0a46ce00 x1855007923803904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076969.509528:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923803904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076969.509540:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076969.509544:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076969.509547:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923803904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076969.509557:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076969.509623:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076969.509642:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076969/real 1769076969] req@ffff969e0a46df80 x1855007923804032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076969.509657:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923804032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076969.509668:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076969.509672:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076969.509690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076969.509701:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076969.509732:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076969.509748:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076969/real 1769076969] req@ffff969e0a46ca80 x1855007923804160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076969.509764:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923804160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076969.509775:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076969.509780:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076969.509785:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076969.828711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076970.533008:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076970.852708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076971.556786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076971.876685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076972.580750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076972.900733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076973.604808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076973.924698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769076974.628744:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076974.628759:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076974.628765:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076974.628778:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3663 00000100:00080000:0.0:1769076974.628794:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076974.628818:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076974.628864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076974.628875:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923804288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076974.628935:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076974.628938:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076974.628940:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076974.628953:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3663 00000100:00080000:0.0:1769076974.628958:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076974.628981:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076974.628994:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076974.629004:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923804288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769076974.629012:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923804416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076974.629025:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076974.629028:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076974.629030:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076974.629034:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3663 00000100:00080000:0.0:1769076974.629038:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076974.629041:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076974.629045:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076974.629057:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923804544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076974.629065:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076974.629069:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769076974.629071:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076974.629073:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3663 00000100:00080000:0.0:1769076974.629094:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076974.629099:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076974.629104:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076974.629120:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923804672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076974.629120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769076974.629132:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076974.629136:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769076974.629137:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076974.629141:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3663 00000100:00080000:0.0:1769076974.629145:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076974.629149:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769076974.629153:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769076974.629162:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923804800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076974.629173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076974/real 1769076974] req@ffff969e06a61880 x1855007923804288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076974.629195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923804288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076974.629206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076974.629220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076974.629232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923804288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076974.629260:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076974.629285:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076974.629325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076974/real 1769076974] req@ffff969e06a63100 x1855007923804416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076974.629343:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923804416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076974.629355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076974.629360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076974.629364:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076974.629370:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076974.629391:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076974.629401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076974/real 1769076974] req@ffff969e06a61f80 x1855007923804544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076974.629412:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923804544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076974.629419:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076974.629421:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076974.629424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076974.629429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076974.629443:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076974.629450:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076974/real 1769076974] req@ffff969e06a62680 x1855007923804672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076974.629459:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923804672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076974.629473:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076974.629476:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076974.629479:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076974.629484:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076974.629501:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076974.629509:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076974/real 1769076974] req@ffff969e06a61500 x1855007923804800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076974.629518:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923804800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076974.629525:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076974.629528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076974.629530:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923804800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076974.948742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076975.652759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076975.972689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076976.676777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076976.996712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076977.700719:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076978.020757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076978.724780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076979.044799:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076979.748823:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076979.748863:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076979.748867:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076979.748891:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3668 00010000:00010000:3.0:1769076979.748903:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076979.748907:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076979.748931:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076979.748962:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923804928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076979.748988:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076979.748991:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076979.748992:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076979.748995:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3668 00000100:00080000:2.0:1769076979.748998:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076979.749001:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076979.749010:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076979.749021:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923805056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076979.749029:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076979.749036:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769076979.749038:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923804928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076979.749038:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076979.749054:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3668 00000100:00080000:2.0:1769076979.749057:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076979.749061:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076979.749064:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076979.749075:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923805184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076979.749088:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076979.749091:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076979.749093:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076979.749096:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3668 00000100:00080000:2.0:1769076979.749101:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076979.749105:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076979.749110:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076979.749118:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923805312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076979.749124:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076979.749127:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076979.749129:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076979.749134:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3668 00000800:00000100:1.0:1769076979.749143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076979.749145:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076979.749150:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076979.749154:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076979.749164:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923805440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076979.749202:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076979/real 1769076979] req@ffff969e0a46ea00 x1855007923804928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076979.749228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923804928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076979.749240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076979.749245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076979.749250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923804928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769076979.749299:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076979.749328:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076979.749343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076979/real 1769076979] req@ffff969e0a46f100 x1855007923805056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076979.749356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923805056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076979.749366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076979.749369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076979.749373:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076979.749380:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076979.749422:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076979.749438:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076979/real 1769076979] req@ffff969e0a46d500 x1855007923805184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076979.749452:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923805184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076979.749459:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076979.749462:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076979.749465:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076979.749471:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076979.749491:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076979.749500:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076979/real 1769076979] req@ffff969e0a46f480 x1855007923805312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076979.749518:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923805312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076979.749526:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076979.749529:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076979.749532:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076979.749537:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076979.749553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076979.749561:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076979/real 1769076979] req@ffff969e0a46d880 x1855007923805440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076979.749571:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923805440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076979.749578:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076979.749580:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076979.749582:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076980.068715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076980.772711:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076981.092688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076981.796799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076982.116694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076982.820791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076983.140790:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076983.844798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076984.164729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076984.868764:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076984.868778:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076984.868784:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076984.868853:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3674 00000100:00080000:2.0:1769076984.868865:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076984.868874:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076984.868874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076984.868938:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923805568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076984.868970:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076984.868975:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076984.868977:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076984.868984:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3674 00000100:00080000:2.0:1769076984.869007:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076984.869014:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076984.869028:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076984.869031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923805568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076984.869068:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923805696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076984.869088:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076984.869094:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076984.869097:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769076984.869105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076984.869107:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3674 00000100:00080000:2.0:1769076984.869119:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076984.869143:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076984.869150:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076984.869159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076984/real 1769076984] req@ffff969e0a46f800 x1855007923805568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076984.869177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923805568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076984.869188:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923805824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076984.869235:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076984.869248:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076984.869253:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076984.869255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076984.869277:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3674 00000100:00080000:1.0:1769076984.869281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076984.869296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923805568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076984.869299:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076984.869308:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076984.869316:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076984.869323:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076984.869347:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923805952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076984.869361:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769076984.869364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076984.869369:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769076984.869373:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769076984.869378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076984/real 1769076984] req@ffff969e0a46c380 x1855007923805696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076984.869380:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3674 00000100:00080000:2.0:1769076984.869391:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769076984.869396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923805696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076984.869400:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076984.869408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076984.869412:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076984.869413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076984.869419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076984.869425:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769076984.869434:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923806080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769076984.869440:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076984.869451:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076984/real 1769076984] req@ffff969e0a46d880 x1855007923805824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076984.869461:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923805824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076984.869467:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076984.869470:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076984.869472:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076984.869492:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076984.869507:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076984.869514:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076984/real 1769076984] req@ffff969e0a46f480 x1855007923805952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076984.869522:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923805952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076984.869528:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076984.869530:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076984.869532:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923805952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076984.869536:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076984.869545:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076984.869550:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076984/real 1769076984] req@ffff969e0a46d500 x1855007923806080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076984.869557:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923806080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076984.869563:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076984.869565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076984.869567:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076985.188672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076985.892981:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076986.212670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076986.916788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076987.236685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076987.940815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076988.260716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076988.964777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076989.284778:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076989.988808:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076989.988824:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076989.988832:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076989.988873:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3679 00000100:00080000:2.0:1769076989.988886:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076989.988894:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769076989.988900:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076989.988935:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923806208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076989.988984:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076989.988991:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076989.988995:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076989.989002:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3679 00000100:00080000:2.0:1769076989.989009:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076989.989016:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076989.989031:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076989.989052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923806208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076989.989052:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923806336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076989.989070:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076989.989076:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076989.989079:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076989.989085:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3679 00000100:00080000:2.0:1769076989.989090:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076989.989111:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076989.989135:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769076989.989147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769076989.989149:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923806464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076989.989160:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076989.989164:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076989.989167:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076989.989171:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3679 00000100:00080000:2.0:1769076989.989176:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076989.989190:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769076989.989195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076989/real 1769076989] req@ffff969e0a46f100 x1855007923806208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769076989.989196:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076989.989210:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923806592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076989.989221:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076989.989225:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076989.989228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923806208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076989.989228:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076989.989232:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3679 00000100:00080000:2.0:1769076989.989236:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076989.989240:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076989.989242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769076989.989246:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769076989.989247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076989.989252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923806208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076989.989257:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923806720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076989.989298:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076989.989339:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076989.989373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076989/real 1769076989] req@ffff969e0a46ea00 x1855007923806336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076989.989384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923806336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076989.989391:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076989.989394:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076989.989397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076989.989402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076989.989418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076989.989425:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076989/real 1769076989] req@ffff969e0a46ca80 x1855007923806464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076989.989434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923806464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076989.989452:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076989.989455:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076989.989458:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076989.989462:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076989.989495:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076989.989507:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076989/real 1769076989] req@ffff969e0a46df80 x1855007923806592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076989.989520:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923806592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076989.989530:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076989.989533:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076989.989536:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076989.989542:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076989.989559:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076989.989569:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076989/real 1769076989] req@ffff969e0a46ce00 x1855007923806720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076989.989582:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923806720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076989.989591:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076989.989594:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076989.989605:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076990.308720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076991.012865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076991.332776:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076992.036815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076992.356697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076993.061986:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076993.380749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076994.085274:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076994.404770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769076995.109235:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076995.109251:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076995.109258:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076995.109273:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3684 00000100:00080000:2.0:1769076995.109289:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076995.109300:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769076995.109353:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923806848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769076995.109365:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076995.109543:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076995.109550:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076995.109553:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076995.109560:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3684 00000100:00080000:2.0:1769076995.109567:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076995.109574:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076995.109587:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076995.109655:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923806976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076995.109672:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076995.109677:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076995.109680:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076995.109687:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3684 00000100:00100000:1.0:1769076995.109696:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923806848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769076995.109697:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076995.109720:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076995.109729:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076995.109752:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923807104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076995.109769:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076995.109775:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769076995.109778:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076995.109784:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3684 00000100:00080000:2.0:1769076995.109792:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769076995.109793:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769076995.109805:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769076995.109815:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769076995.109845:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923807232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076995.109849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076995/real 1769076995] req@ffff969e0a46e300 x1855007923806848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076995.109888:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076995.109895:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769076995.109898:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923806848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076995.109900:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076995.109914:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3684 00000100:00080000:2.0:1769076995.109925:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076995.109933:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076995.109935:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769076995.109947:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769076995.109949:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769076995.109960:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923806848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769076995.109976:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923807360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769076995.109979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076995.110035:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076995.110081:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076995/real 1769076995] req@ffff969e0a46ed80 x1855007923806976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076995.110101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923806976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076995.110125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076995.110131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076995.110137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923806976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076995.110179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769076995.110216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076995.110257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076995/real 1769076995] req@ffff969e0a46ce00 x1855007923807104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076995.110278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923807104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076995.110291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076995.110416:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076995.110423:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769076995.110434:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076995.110471:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076995.110490:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076995/real 1769076995] req@ffff969e0a46df80 x1855007923807232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076995.110510:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923807232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076995.110524:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076995.110528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076995.110532:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769076995.110538:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769076995.110560:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769076995.110571:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076995/real 1769076995] req@ffff969e0a46ca80 x1855007923807360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769076995.110582:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923807360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769076995.110648:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769076995.110652:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769076995.110656:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769076995.428702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076996.132994:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076996.452759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076997.156929:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076997.476711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076998.180835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076998.500736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076999.204788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076999.524712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077000.228767:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077000.228779:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077000.228786:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077000.228798:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3689 00000100:00080000:0.0:1769077000.228812:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077000.228822:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077000.228826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077000.228869:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923807488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077000.228906:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077000.228933:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077000.228936:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077000.228942:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3689 00000100:00080000:0.0:1769077000.228948:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077000.228954:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077000.228963:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923807488:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077000.228970:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077000.228992:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923807616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077000.229008:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077000.229012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077000.229015:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077000.229020:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3689 00000100:00080000:0.0:1769077000.229035:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077000.229040:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077000.229047:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077000.229049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077000.229065:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923807744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077000.229080:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077000.229084:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077000.229105:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077000.229111:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3689 00000100:00080000:0.0:1769077000.229117:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077000.229125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077000/real 1769077000] req@ffff969e06a61180 x1855007923807488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077000.229126:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077000.229150:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077000.229168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923807488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077000.229203:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923807872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077000.229206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077000.229225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769077000.229229:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077000.229235:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077000.229241:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077000.229243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923807488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077000.229249:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3689 00000100:00080000:0.0:1769077000.229258:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077000.229282:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077000.229289:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077000.229303:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923808000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077000.229318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077000.229366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077000.229403:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077000/real 1769077000] req@ffff969e06a60380 x1855007923807616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077000.229418:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923807616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077000.229429:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077000.229433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077000.229438:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077000.229460:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077000.229484:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077000.229496:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077000/real 1769077000] req@ffff969e06a60a80 x1855007923807744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077000.229509:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923807744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077000.229518:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077000.229522:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077000.229525:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077000.229532:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077000.229637:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077000.229658:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077000/real 1769077000] req@ffff969e06a60700 x1855007923807872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077000.229683:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923807872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077000.229717:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077000.229724:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077000.229731:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923807872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077000.229763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077000.229803:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077000.229847:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077000/real 1769077000] req@ffff969e06a63800 x1855007923808000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077000.229900:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923808000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077000.229913:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077000.229917:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077000.229921:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077000.548712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077001.252805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077001.572770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077002.276709:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077002.596753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077003.300757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077003.620698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077004.324765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077004.644688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077005.348675:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077005.348684:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077005.348688:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077005.348697:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3694 00000100:00080000:0.0:1769077005.348706:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077005.348712:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077005.348732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077005.348746:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923808128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077005.348769:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077005.348772:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077005.348773:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077005.348776:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3694 00000100:00080000:0.0:1769077005.348794:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077005.348798:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077005.348804:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923808128:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077005.348808:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077005.348821:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923808256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077005.348830:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077005.348832:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077005.348834:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077005.348836:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3694 00000100:00080000:0.0:1769077005.348839:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077005.348842:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077005.348845:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077005.348854:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923808384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077005.348860:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077005.348863:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077005.348864:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077005.348867:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3694 00000100:00080000:0.0:1769077005.348869:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077005.348871:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077005.348874:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077005.348879:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077005.348883:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923808512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077005.348900:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077005.348903:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077005.348905:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077005.348908:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3694 00000100:00080000:0.0:1769077005.348912:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077005.348917:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077005/real 1769077005] req@ffff969e06a60000 x1855007923808128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077005.348920:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077005.348927:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077005.348939:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923808640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077005.348942:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923808128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077005.348965:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077005.348969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077005.348973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923808128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077005.348993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077005.349010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077005.349018:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077005/real 1769077005] req@ffff969e06a62d80 x1855007923808256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077005.349026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923808256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077005.349043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077005.349046:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077005.349048:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077005.349053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077005.349067:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077005.349073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077005/real 1769077005] req@ffff969e06a63800 x1855007923808384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077005.349081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923808384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077005.349086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077005.349088:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077005.349090:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077005.349094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077005.349104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077005.349110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077005/real 1769077005] req@ffff969e06a60700 x1855007923808512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077005.349117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923808512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077005.349123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077005.349124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077005.349133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077005.349154:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077005.349171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077005.349179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077005/real 1769077005] req@ffff969e06a60a80 x1855007923808640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077005.349189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923808640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077005.349196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077005.349198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077005.349201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077005.668687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077006.372880:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077006.692725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077007.396851:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077007.716677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077008.420809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077008.740698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077009.444828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077009.764695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077010.468701:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077010.468710:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077010.468714:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077010.468723:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3699 00000100:00080000:2.0:1769077010.468732:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769077010.468736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077010.468739:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077010.468773:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923808768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077010.468796:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077010.468798:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077010.468800:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077010.468803:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3699 00000100:00080000:2.0:1769077010.468805:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077010.468809:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077010.468827:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923808768:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077010.468833:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077010.468847:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923808896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077010.468856:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077010.468858:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077010.468860:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077010.468863:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3699 00000100:00080000:2.0:1769077010.468866:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077010.468869:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077010.468872:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077010.468879:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923809024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077010.468888:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077010.468888:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077010.468894:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077010.468896:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077010.468900:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3699 00000100:00080000:2.0:1769077010.468903:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077010.468906:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077010.468909:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077010.468919:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923809152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077010.468924:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077010/real 1769077010] req@ffff969e0a46ea00 x1855007923808768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077010.468929:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077010.468936:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077010.468943:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077010.468948:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3699 00000100:00100000:1.0:1769077010.468950:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923808768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077010.468953:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077010.468959:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077010.468962:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077010.468973:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077010.468974:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923809280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077010.468982:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077010.468990:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923808768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077010.469011:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077010.469027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077010.469043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077010/real 1769077010] req@ffff969e0a46f100 x1855007923808896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077010.469052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923808896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077010.469058:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077010.469061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077010.469063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923808896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077010.469067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077010.469079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077010.469085:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077010/real 1769077010] req@ffff969e0a46d500 x1855007923809024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077010.469093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923809024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077010.469098:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077010.469100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077010.469102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077010.469106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077010.469128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077010.469133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077010/real 1769077010] req@ffff969e0a46f480 x1855007923809152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077010.469140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923809152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077010.469150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077010.469152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077010.469155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077010.469159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077010.469171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077010.469177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077010/real 1769077010] req@ffff969e0a46d880 x1855007923809280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077010.469184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923809280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077010.469190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077010.469198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077010.469200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077010.788734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077011.492810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077011.812774:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077012.516809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077012.836750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077013.540851:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077013.860814:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077014.564818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077014.884804:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077015.588726:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077015.588737:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077015.588743:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077015.588754:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3704 00000100:00080000:3.0:1769077015.588767:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077015.588771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077015.588775:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077015.588819:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923809408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077015.588845:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077015.588848:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077015.588849:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077015.588853:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3704 00000100:00080000:3.0:1769077015.588856:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077015.588881:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077015.588889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923809408:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077015.588896:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077015.588921:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923809536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077015.588932:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077015.588936:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077015.588937:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077015.588940:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3704 00000100:00080000:3.0:1769077015.588945:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077015.588949:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077015.588954:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077015.588961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077015.588967:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923809664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077015.588978:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077015.588981:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077015.588982:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077015.588985:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3704 00000100:00080000:3.0:1769077015.588988:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077015.588993:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077015.588999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077015/real 1769077015] req@ffff969e0a7fd180 x1855007923809408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077015.589001:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077015.589019:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923809792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077015.589028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923809408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077015.589033:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077015.589041:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077015.589042:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077015.589048:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077015.589052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077015.589059:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3704 00000100:00100000:1.0:1769077015.589062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923809408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077015.589069:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077015.589076:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077015.589089:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077015.589108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077015.589110:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923809920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077015.589137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077015.589150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077015/real 1769077015] req@ffff969e0a7fd880 x1855007923809536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077015.589161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923809536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077015.589167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077015.589170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077015.589172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077015.589176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077015.589187:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077015.589193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077015/real 1769077015] req@ffff969e0a7fca80 x1855007923809664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077015.589200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923809664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077015.589205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077015.589207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077015.589209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077015.589213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077015.589223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077015.589229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077015/real 1769077015] req@ffff969e0a7fe300 x1855007923809792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077015.589243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923809792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077015.589248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077015.589251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077015.589253:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077015.589265:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077015.589296:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077015.589305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077015/real 1769077015] req@ffff969e0a7fc380 x1855007923809920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077015.589313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923809920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077015.589319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077015.589321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077015.589323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923809920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077015.908755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077016.612897:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077016.932800:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077017.636799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077017.956723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077018.660867:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077018.980659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077019.684725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077020.004709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077020.708715:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077020.708730:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077020.708737:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077020.708766:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3709 00000100:00080000:3.0:1769077020.708781:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077020.708790:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077020.708845:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923810048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769077020.708865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077020.708887:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077020.708892:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077020.708895:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077020.708900:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3709 00000100:00080000:3.0:1769077020.708915:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077020.708920:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077020.708932:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077020.708945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923810048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077020.708950:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923810176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077020.708986:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077020.708990:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077020.708992:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077020.708997:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3709 00000100:00080000:3.0:1769077020.709003:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077020.709007:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077020.709014:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077020.709025:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077020.709041:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923810304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077020.709053:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077020.709058:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077020.709060:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077020.709066:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3709 00000100:00100000:1.0:1769077020.709071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077020/real 1769077020] req@ffff969e0a7fdf80 x1855007923810048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077020.709073:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077020.709086:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077020.709098:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077020.709108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923810048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077020.709116:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923810432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077020.709134:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077020.709140:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077020.709144:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077020.709152:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3709 00000100:00080000:3.0:1769077020.709166:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077020.709176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077020.709185:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077020.709187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:3.0:1769077020.709217:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077020.709243:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923810048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077020.709252:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923810560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077020.709271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077020.709294:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077020.709304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077020/real 1769077020] req@ffff969e0a7fea00 x1855007923810176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077020.709313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923810176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077020.709320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077020.709322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077020.709325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077020.709329:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077020.709340:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077020.709347:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077020/real 1769077020] req@ffff969e0a7fc700 x1855007923810304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077020.709355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923810304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077020.709361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077020.709363:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077020.709365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077020.709376:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077020.709391:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077020.709398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077020/real 1769077020] req@ffff969e0a7fce00 x1855007923810432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077020.709406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923810432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077020.709413:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077020.709414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077020.709417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077020.709421:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077020.709431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077020.709444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077020/real 1769077020] req@ffff969e0a7fe680 x1855007923810560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077020.709453:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923810560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077020.709460:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077020.709462:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077020.709464:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077021.028714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077021.732784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077022.052732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077022.756810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077023.076672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077023.780859:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077024.100728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077024.804790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077025.124745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077025.828686:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077025.828706:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077025.828710:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077025.828719:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3715 00000100:00080000:3.0:1769077025.828729:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077025.828735:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077025.828772:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923810688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077025.828793:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077025.828796:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077025.828798:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077025.828802:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3715 00010000:00010000:0.0:1769077025.828805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077025.828805:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077025.828810:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077025.828819:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077025.828829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923810688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077025.828832:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923810816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077025.828844:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077025.828860:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077025.828862:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077025.828865:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3715 00000100:00080000:3.0:1769077025.828868:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077025.828871:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077025.828875:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077025.828887:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923810944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077025.828891:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077025.828900:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077025.828910:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077025.828912:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077025.828916:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3715 00000100:00080000:3.0:1769077025.828919:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077025.828922:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077025.828926:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077025.828928:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077025/real 1769077025] req@ffff969e0a7fed80 x1855007923810688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077025.828936:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923811072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077025.828970:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077025.828973:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077025.828974:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077025.828977:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3715 00000100:00080000:3.0:1769077025.828979:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077025.828984:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077025.828986:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923810688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077025.828998:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077025.829001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077025.829007:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077025.829010:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923811200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077025.829015:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923810688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077025.829043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077025.829064:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077025.829083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077025/real 1769077025] req@ffff969e0a7fe680 x1855007923810816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077025.829092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923810816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077025.829098:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077025.829100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077025.829102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077025.829135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077025.829151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077025.829159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077025/real 1769077025] req@ffff969e0a7fce00 x1855007923810944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077025.829167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923810944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077025.829189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077025.829192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077025.829195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923810944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077025.829199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077025.829215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077025.829222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077025/real 1769077025] req@ffff969e0a7fc700 x1855007923811072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077025.829229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923811072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077025.829235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077025.829237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077025.829239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077025.829242:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077025.829252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077025.829257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077025/real 1769077025] req@ffff969e0a7fea00 x1855007923811200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077025.829264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923811200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077025.829270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077025.829271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077025.829278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077026.148699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077026.852764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077027.172672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077027.876734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077028.196695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077028.900791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077029.220724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077029.924754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077030.244695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077030.948729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077030.948750:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077030.948777:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077030.948805:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077030.948816:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3720 00000100:00080000:0.0:1769077030.948846:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077030.948854:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077030.948893:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923811328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077030.948945:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077030.948948:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077030.948950:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077030.948954:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3720 00000100:00080000:0.0:1769077030.948957:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077030.948960:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077030.948968:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077030.948978:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923811456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077030.948987:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077030.948990:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077030.948992:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077030.948995:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3720 00000100:00080000:0.0:1769077030.948997:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077030.949000:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077030.949005:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077030.949010:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923811328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077030.949051:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923811584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077030.949075:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077030.949078:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077030.949081:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077030.949084:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3720 00000100:00080000:0.0:1769077030.949097:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077030.949101:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077030.949106:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077030.949117:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923811712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077030.949125:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077030.949128:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077030.949130:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077030.949133:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3720 00000100:00080000:0.0:1769077030.949137:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077030.949141:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077030.949145:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077030.949150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077030.949156:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923811840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077030.949216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077030.949247:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077030.949270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077031.268701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077031.972846:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077032.292728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077032.996737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077033.316740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077034.020866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077034.020900:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769077034.020947:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077034.020963:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077034.021023:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077034.021029:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077034.021035:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077034.021041:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077034.021067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077030/real 1769077034] req@ffff969e06a61500 x1855007923811584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.021088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923811584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.021097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.021101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077034.021104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077034.021109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077030/real 1769077034] req@ffff969e06a62680 x1855007923811712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.021117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923811712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.021123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.021125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077034.021128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077034.021146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077030/real 1769077034] req@ffff969e06a61f80 x1855007923811840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.021159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923811840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.021175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.021179:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077034.021182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077034.021190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077030/real 1769077034] req@ffff969e06a60380 x1855007923811328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.021204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923811328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.021212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.021226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077034.021230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923811328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077034.021237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077030/real 1769077034] req@ffff969e06a61180 x1855007923811456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077034.021250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923811456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077034.021260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077034.021265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077034.021267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923811456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077034.340684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077035.044772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077035.364652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077036.068737:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077036.068746:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077036.068749:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077036.068759:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3725 00000100:00080000:2.0:1769077036.068780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077036.068786:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077036.068807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077036.068820:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923811968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077036.068846:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077036.068849:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077036.068850:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077036.068864:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3725 00000100:00080000:2.0:1769077036.068866:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077036.068869:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077036.068877:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077036.068885:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923812096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077036.068915:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077036.068918:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077036.068920:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077036.068925:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3725 00000100:00100000:1.0:1769077036.068960:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923811968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077036.068962:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077036.068979:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077036.068991:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077036.069014:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923812224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077036.069025:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077036.069029:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077036.069031:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077036.069035:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3725 00000100:00080000:2.0:1769077036.069040:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077036.069047:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077036.069053:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077036.069065:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923812352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077036.069075:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077036.069080:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077036.069082:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077036.069089:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3725 00000800:00000100:1.0:1769077036.069093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077036.069105:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077036.069110:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077036.069116:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077036.069128:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923812480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077036.069170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077036/real 1769077036] req@ffff969e0a46c380 x1855007923811968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077036.069203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923811968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077036.069222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077036.069248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077036.069256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923811968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077036.069300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077036.069346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077036.069367:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077036/real 1769077036] req@ffff969e0a46f800 x1855007923812096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077036.069387:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923812096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077036.069402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077036.069409:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077036.069415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077036.069426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077036.069461:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077036.069479:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077036/real 1769077036] req@ffff969e0a46d880 x1855007923812224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077036.069519:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923812224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077036.069534:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077036.069539:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077036.069545:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077036.069556:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077036.069641:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077036.069665:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077036/real 1769077036] req@ffff969e0a46f480 x1855007923812352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077036.069682:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923812352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077036.069694:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077036.069699:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077036.069704:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077036.069713:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077036.069741:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077036.069758:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077036/real 1769077036] req@ffff969e0a46d500 x1855007923812480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077036.069774:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923812480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077036.069805:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077036.069810:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077036.069816:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077036.388665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077037.092799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077037.412690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077038.116889:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077038.436715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077039.140758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077039.460763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077040.164821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077040.484770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077041.188683:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077041.188696:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077041.188701:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077041.188713:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3730 00000100:00080000:2.0:1769077041.188726:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077041.188736:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077041.188766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077041.188792:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923812608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077041.188836:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077041.188841:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077041.188844:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077041.188862:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3730 00000100:00080000:2.0:1769077041.188867:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077041.188873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923812608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077041.188877:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077041.188892:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077041.188929:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923812736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077041.188939:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077041.188941:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077041.188943:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077041.188947:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3730 00000100:00080000:2.0:1769077041.188950:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077041.188953:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077041.188957:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077041.188963:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077041.188968:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923812864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077041.188976:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077041.188979:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077041.188980:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077041.188983:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3730 00000100:00080000:2.0:1769077041.188986:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077041.188989:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077041.188992:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077041.188997:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923812992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077041.189003:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077041.189006:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077041.189008:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077041.189016:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3730 00000100:00080000:2.0:1769077041.189020:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077041.189023:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077041.189028:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077041.189029:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077041/real 1769077041] req@ffff969e0a46f100 x1855007923812608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077041.189041:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923813120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077041.189054:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923812608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077041.189065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077041.189069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077041.189072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923812608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077041.189099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077041.189118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077041.189127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077041/real 1769077041] req@ffff969e0a46ea00 x1855007923812736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077041.189135:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923812736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077041.189141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077041.189144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077041.189146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077041.189159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077041.189174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077041.189181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077041/real 1769077041] req@ffff969e0a46ca80 x1855007923812864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077041.189189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923812864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077041.189195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077041.189197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077041.189198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077041.189203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077041.189235:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077041.189242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077041/real 1769077041] req@ffff969e0a46df80 x1855007923812992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077041.189252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923812992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077041.189259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077041.189261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077041.189263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923812992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077041.189267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077041.189279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077041.189286:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077041/real 1769077041] req@ffff969e0a46ce00 x1855007923813120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077041.189302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923813120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077041.189310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077041.189312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077041.189315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077041.508810:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077042.212901:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077042.532739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077043.236786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077043.556795:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077044.260749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077044.580829:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077045.284794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077045.604717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077046.308702:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077046.308711:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077046.308715:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077046.308724:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3735 00010000:00010000:0.0:1769077046.308734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077046.308736:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077046.308746:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077046.308803:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923813248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077046.308825:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077046.308828:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077046.308841:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077046.308846:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3735 00000100:00080000:3.0:1769077046.308850:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077046.308854:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077046.308863:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077046.308865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923813248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077046.308875:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923813376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077046.308885:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077046.308887:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077046.308889:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077046.308892:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3735 00000100:00080000:3.0:1769077046.308894:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077046.308897:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077046.308900:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077046.308907:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923813504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077046.308914:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077046.308917:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077046.308918:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077046.308920:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3735 00000100:00080000:3.0:1769077046.308923:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077046.308927:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077046.308930:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077046.308935:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077046.308950:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923813632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077046.308957:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077046.308960:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077046.308962:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077046.308963:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077046/real 1769077046] req@ffff969e0a7fdf80 x1855007923813248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077046.308967:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3735 00000100:00080000:3.0:1769077046.308972:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077046.308978:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077046.308983:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077046.308987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923813248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077046.308993:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923813760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077046.309001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077046.309008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077046.309014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923813248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077046.309040:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077046.309063:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077046.309077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077046/real 1769077046] req@ffff969e0a7fc380 x1855007923813376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077046.309086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923813376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077046.309102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077046.309105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077046.309107:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077046.309112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077046.309126:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077046.309143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077046/real 1769077046] req@ffff969e0a7fe300 x1855007923813504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077046.309151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923813504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077046.309156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077046.309158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077046.309161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077046.309164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077046.309176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077046.309182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077046/real 1769077046] req@ffff969e0a7fca80 x1855007923813632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077046.309189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923813632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077046.309194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077046.309196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077046.309204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077046.309208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077046.309221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077046.309228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077046/real 1769077046] req@ffff969e0a7fd880 x1855007923813760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077046.309235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923813760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077046.309240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077046.309242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077046.309244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923813760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077046.628733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077047.332782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077047.652717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077048.356776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077048.676718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077049.380889:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077049.700719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077050.404799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077050.724729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077051.428731:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077051.428747:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1769077051.428749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077051.428754:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077051.428767:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3740 00000100:00080000:0.0:1769077051.428779:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077051.428792:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077051.428837:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923813888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077051.428861:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077051.428865:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077051.428867:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077051.428872:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3740 00000100:00080000:0.0:1769077051.428877:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077051.428882:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077051.428893:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077051.428899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923813888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077051.428906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923814016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077051.428925:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077051.428928:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077051.428930:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077051.428934:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3740 00000100:00080000:0.0:1769077051.428937:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077051.428941:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077051.428946:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077051.428979:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923814144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077051.428980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077051.428989:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077051.428992:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077051.428994:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077051.428998:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3740 00000100:00080000:0.0:1769077051.429001:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077051.429006:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077051.429011:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077051.429022:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923814272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077051.429024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077051/real 1769077051] req@ffff969e06a63100 x1855007923813888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077051.429032:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077051.429035:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077051.429037:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077051.429040:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3740 00000100:00080000:0.0:1769077051.429043:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077051.429045:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923813888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077051.429047:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077051.429052:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077051.429057:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077051.429061:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923814400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077051.429062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077051.429067:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923813888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077051.429096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077051.429120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077051.429139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077051/real 1769077051] req@ffff969e06a61880 x1855007923814016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077051.429156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923814016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077051.429163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077051.429165:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077051.429168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077051.429172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077051.429187:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077051.429195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077051/real 1769077051] req@ffff969e06a61180 x1855007923814144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077051.429203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923814144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077051.429208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077051.429211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077051.429213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077051.429217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077051.429227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077051.429233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077051/real 1769077051] req@ffff969e06a60380 x1855007923814272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077051.429241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923814272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077051.429251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077051.429254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077051.429256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077051.429262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077051.429292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077051.429304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077051/real 1769077051] req@ffff969e06a61f80 x1855007923814400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077051.429317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923814400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077051.429326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077051.429336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077051.429338:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077051.748663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077052.452823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077052.772784:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077053.476872:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077053.796766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077054.500754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077054.820741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077055.524817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077055.844839:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077056.548765:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077056.548777:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077056.548783:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077056.548813:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3745 00000100:00080000:0.0:1769077056.548826:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077056.548834:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077056.548867:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077056.548875:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923814528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077056.548914:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077056.548918:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077056.548920:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077056.548925:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3745 00000100:00080000:0.0:1769077056.548929:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077056.548933:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077056.548943:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077056.548958:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923814656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077056.548971:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077056.548977:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077056.548981:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077056.548981:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923814528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077056.549021:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3745 00000100:00080000:0.0:1769077056.549024:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077056.549028:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077056.549033:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077056.549044:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923814784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077056.549063:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077056.549066:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077056.549068:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077056.549072:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3745 00000100:00080000:0.0:1769077056.549089:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077056.549094:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077056.549098:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077056.549121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077056.549128:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923814912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077056.549138:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077056.549141:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077056.549143:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077056.549149:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3745 00000100:00080000:0.0:1769077056.549153:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077056.549158:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077056.549164:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077056.549177:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923815040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077056.549187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077056/real 1769077056] req@ffff969e06a62680 x1855007923814528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077056.549215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923814528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077056.549230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077056.549237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077056.549241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923814528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077056.549295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077056.549324:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077056.549345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077056/real 1769077056] req@ffff969e06a61500 x1855007923814656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077056.549367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923814656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077056.549381:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077056.549388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077056.549394:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077056.549405:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077056.549442:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077056.549462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077056/real 1769077056] req@ffff969e06a60a80 x1855007923814784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077056.549495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923814784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077056.549510:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077056.549516:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077056.549522:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077056.549534:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077056.549573:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077056.549682:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077056/real 1769077056] req@ffff969e06a60700 x1855007923814912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077056.549712:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923814912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077056.549724:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077056.549728:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077056.549732:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923814912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077056.549751:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077056.549781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077056.549795:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077056/real 1769077056] req@ffff969e06a63800 x1855007923815040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077056.549809:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923815040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077056.549819:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077056.549823:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077056.549826:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077056.868749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077057.572809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077057.892764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077058.597025:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077058.916710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077059.620792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077059.940713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077060.644946:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077060.964709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077061.669252:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077061.669817:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077061.669827:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077061.669832:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077061.669843:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3750 00000100:00080000:1.0:1769077061.669856:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077061.669864:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077061.669909:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923815168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077061.669957:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077061.669962:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077061.669965:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077061.669971:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3750 00000100:00080000:1.0:1769077061.669977:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077061.669983:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077061.670000:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077061.670016:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923815296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077061.670027:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077061.670030:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077061.670032:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077061.670039:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3750 00000100:00100000:0.0:1769077061.670041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923815168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077061.670045:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077061.670055:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077061.670062:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077061.670079:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923815424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077061.670092:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077061.670096:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077061.670098:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077061.670102:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3750 00000100:00080000:1.0:1769077061.670116:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077061.670125:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:0.0:1769077061.670127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:1.0:1769077061.670148:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077061.670168:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923815552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077061.670180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077061/real 1769077061] req@ffff969e10b61500 x1855007923815168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077061.670184:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077061.670194:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077061.670200:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077061.670211:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3750 00000100:00080000:1.0:1769077061.670217:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077061.670224:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923815168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077061.670225:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077061.670237:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769077061.670246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077061.670257:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923815680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077061.670258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077061.670267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923815168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077061.670333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077061.670369:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077061.670382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077061/real 1769077061] req@ffff969e10b61f80 x1855007923815296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077061.670393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923815296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077061.670400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077061.670403:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077061.670406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077061.670411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077061.670426:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077061.670433:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077061/real 1769077061] req@ffff969e10b60a80 x1855007923815424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077061.670442:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923815424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077061.670460:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077061.670463:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077061.670466:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077061.670471:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077061.670486:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077061.670493:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077061/real 1769077061] req@ffff969e10b62a00 x1855007923815552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077061.670520:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923815552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077061.670539:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077061.670541:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077061.670544:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077061.670549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077061.670566:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077061.670574:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077061/real 1769077061] req@ffff969e10b62680 x1855007923815680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077061.670583:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923815680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077061.670626:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077061.670632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077061.670663:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077061.988668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077062.692992:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077063.012682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077063.716914:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077064.036754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077064.740763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077065.060716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077065.764804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077066.084699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077066.788743:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077066.788754:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077066.788758:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077066.788766:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3755 00000100:00080000:2.0:1769077066.788776:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077066.788782:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077066.788798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077066.788818:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923815808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077066.788844:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077066.788847:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077066.788848:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077066.788851:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3755 00000100:00080000:2.0:1769077066.788853:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077066.788856:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077066.788864:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077066.788873:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923815936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077066.788882:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077066.788885:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077066.788886:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077066.788890:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3755 00000100:00080000:2.0:1769077066.788896:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077066.788898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923815808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077066.788902:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077066.788925:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077066.788951:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923816064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077066.788958:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077066.788961:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077066.788962:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077066.788965:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3755 00000100:00080000:2.0:1769077066.788967:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077066.788970:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077066.788974:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077066.788981:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923816192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077066.788986:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077066.788989:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077066.788991:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077066.788994:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3755 00000100:00080000:2.0:1769077066.788997:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077066.788999:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077066.789002:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769077066.789010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077066.789012:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923816320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077066.789072:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077066/real 1769077066] req@ffff969e0a46ed80 x1855007923815808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077066.789089:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923815808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077066.789099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077066.789103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077066.789106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923815808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077066.789131:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077066.789148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077066.789156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077066/real 1769077066] req@ffff969e0a46e300 x1855007923815936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077066.789164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923815936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077066.789170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077066.789172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077066.789174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923815936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077066.789178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077066.789189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077066.789210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077066/real 1769077066] req@ffff969e0a46ce00 x1855007923816064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077066.789218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923816064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077066.789224:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077066.789226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077066.789229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077066.789232:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077066.789246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077066.789252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077066/real 1769077066] req@ffff969e0a46df80 x1855007923816192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077066.789260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923816192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077066.789265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077066.789267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077066.789269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077066.789273:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077066.789282:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077066.789288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077066/real 1769077066] req@ffff969e0a46ca80 x1855007923816320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077066.789295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923816320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077066.789306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077066.789308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077066.789311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077067.108689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077067.812834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077068.132764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077068.836725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077069.156790:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077069.860788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077070.180724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077070.884873:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077071.204790:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077071.908722:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077071.908734:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077071.908740:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077071.908751:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3761 00000100:00080000:2.0:1769077071.908764:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077071.908773:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077071.908788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077071.908820:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923816448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077071.908847:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077071.908849:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077071.908851:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077071.908854:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3761 00000100:00080000:2.0:1769077071.908856:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077071.908859:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077071.908869:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077071.908879:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923816576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077071.908887:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077071.908890:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077071.908892:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077071.908895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923816448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077071.908897:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3761 00000100:00080000:2.0:1769077071.908903:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077071.908907:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077071.908930:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077071.908941:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923816704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077071.908948:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077071.908951:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077071.908952:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077071.908955:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3761 00000100:00080000:2.0:1769077071.908958:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077071.908960:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077071.908964:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077071.908972:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923816832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077071.908981:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077071.908982:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077071.908987:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077071.908988:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077071.908992:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3761 00000100:00080000:2.0:1769077071.908996:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077071.908998:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077071.909002:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077071.909009:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923816960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077071.909028:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077071/real 1769077071] req@ffff969e0a46ea00 x1855007923816448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077071.909052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923816448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077071.909064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077071.909069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077071.909074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923816448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077071.909098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077071.909122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077071.909136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077071/real 1769077071] req@ffff969e0a46f100 x1855007923816576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077071.909150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923816576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077071.909161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077071.909165:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077071.909176:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077071.909181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077071.909212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077071.909220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077071/real 1769077071] req@ffff969e0a46d500 x1855007923816704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077071.909229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923816704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077071.909234:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077071.909237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077071.909239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077071.909244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077071.909256:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077071.909261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077071/real 1769077071] req@ffff969e0a46f480 x1855007923816832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077071.909269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923816832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077071.909275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077071.909277:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077071.909279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077071.909282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077071.909292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077071.909306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077071/real 1769077071] req@ffff969e0a46d880 x1855007923816960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077071.909314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923816960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077071.909320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077071.909323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077071.909325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923816960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077072.228725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077072.932761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077073.252715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077073.956743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077074.276726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077074.980812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077075.300690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077076.004748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077076.324698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077077.028736:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077077.028749:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077077.028775:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077077.028794:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3766 00010000:00010000:3.0:1769077077.028800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077077.028808:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077077.028817:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077077.028857:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923817088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077077.028885:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077077.028888:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077077.028890:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077077.028893:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3766 00000100:00080000:2.0:1769077077.028896:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077077.028899:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077077.028908:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077077.028922:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923817216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077077.028930:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077077.028934:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077077.028937:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077077.028942:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3766 00000100:00100000:0.0:1769077077.028951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923817088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077077.028969:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077077.028977:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077077.028984:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077077.029006:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923817344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077077.029035:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769077077.029038:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077077.029045:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077077.029048:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077077.029056:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3766 00000100:00080000:2.0:1769077077.029062:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077077.029067:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077077.029073:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077077.029084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077077/real 1769077077] req@ffff969e0a46f800 x1855007923817088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077077.029090:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923817472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077077.029134:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077077.029149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923817088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077077.029168:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077077.029180:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077077.029182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077077.029189:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3766 00000100:00080000:0.0:1769077077.029193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769077077.029196:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077077.029203:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077077.029205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923817088:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077077.029211:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077077.029231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077077.029276:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923817600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077077.029283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077077.029308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077077/real 1769077077] req@ffff969e0a46c380 x1855007923817216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077077.029328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923817216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077077.029357:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077077.029360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077077.029363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077077.029399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077077.029416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077077.029424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077077/real 1769077077] req@ffff969e0a46d880 x1855007923817344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077077.029431:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923817344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077077.029437:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077077.029439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077077.029441:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077077.029445:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077077.029455:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077077.029460:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077077/real 1769077077] req@ffff969e0a46f480 x1855007923817472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077077.029467:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923817472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077077.029473:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077077.029475:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077077.029482:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077077.029486:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077077.029500:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077077.029506:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077077/real 1769077077] req@ffff969e0a46d500 x1855007923817600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077077.029513:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923817600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077077.029529:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077077.029531:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077077.029533:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077077.348711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077078.052794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077078.372665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077079.076827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077079.396693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077080.100714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077080.420674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077081.124792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077081.444696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077082.148721:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077082.148733:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077082.148738:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077082.148749:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3771 00000100:00080000:2.0:1769077082.148778:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769077082.148782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077082.148790:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077082.148829:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923817728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077082.148861:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077082.148865:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077082.148866:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077082.148871:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3771 00000100:00080000:2.0:1769077082.148874:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077082.148879:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077082.148888:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077082.148903:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923817856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077082.148909:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923817728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077082.148925:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077082.148929:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077082.148932:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077082.148937:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3771 00000100:00080000:2.0:1769077082.148942:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077082.148947:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077082.148952:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077082.148965:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923817984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077082.148972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077082.148985:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077082.148996:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077082.148999:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077082.149004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077082/real 1769077082] req@ffff969e0a46f100 x1855007923817728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077082.149006:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3771 00000100:00080000:2.0:1769077082.149015:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077082.149022:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077082.149027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923817728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077082.149030:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769077082.149041:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077082.149045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077082.149049:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923817728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077082.149053:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923818112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077082.149066:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077082.149070:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1769077082.149072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817856:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077082.149074:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077082.149082:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3771 00000100:00080000:2.0:1769077082.149087:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077082.149094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077082.149094:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077082.149100:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077082.149114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077082/real 1769077082] req@ffff969e0a46ea00 x1855007923817856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077082.149118:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923818240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077082.149135:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923817856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077082.149144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077082.149147:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077082.149150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077082.149154:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077082.149187:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077082.149195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077082/real 1769077082] req@ffff969e0a46ca80 x1855007923817984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077082.149203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923817984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077082.149209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077082.149211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077082.149213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923817984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077082.149217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077082.149228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077082.149234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077082/real 1769077082] req@ffff969e0a46df80 x1855007923818112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077082.149241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923818112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077082.149252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077082.149254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077082.149256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077082.149281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077082.149297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077082.149303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077082/real 1769077082] req@ffff969e0a46ce00 x1855007923818240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077082.149311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923818240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077082.149317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077082.149319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077082.149321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077082.468684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077083.172765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077083.492695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077084.196761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077084.516713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077085.220751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077085.540719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077086.244703:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077086.564689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077087.268699:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077087.268707:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077087.268712:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077087.268720:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3776 00000100:00080000:3.0:1769077087.268730:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077087.268736:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077087.268754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077087.268771:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923818368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077087.268793:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077087.268795:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077087.268797:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077087.268800:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3776 00000100:00080000:3.0:1769077087.268819:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077087.268820:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923818368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077087.268826:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077087.268835:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077087.268848:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923818496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077087.268856:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077087.268859:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077087.268860:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077087.268863:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3776 00000100:00080000:3.0:1769077087.268865:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077087.268868:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077087.268871:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077087.268879:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923818624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077087.268884:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077087.268893:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077087.268896:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077087.268898:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077087.268902:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3776 00000100:00080000:3.0:1769077087.268905:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077087.268907:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077087.268911:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077087.268920:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077087/real 1769077087] req@ffff969e0a7fd180 x1855007923818368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077087.268921:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923818752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077087.268955:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923818368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077087.268961:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077087.268968:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077087.268971:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077087.268976:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3776 00000100:00080000:0.0:1769077087.268977:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077087.268979:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077087.268983:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077087.268986:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077087.268990:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077087.268999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923818368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077087.269004:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923818880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077087.269025:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077087.269044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077087.269052:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077087/real 1769077087] req@ffff969e0a7ffb80 x1855007923818496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077087.269061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923818496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077087.269067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077087.269069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077087.269091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077087.269097:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077087.269112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077087.269120:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077087/real 1769077087] req@ffff969e0a7fdc00 x1855007923818624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077087.269127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923818624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077087.269133:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077087.269135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077087.269137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077087.269141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077087.269151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077087.269164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077087/real 1769077087] req@ffff969e0a7ff800 x1855007923818752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077087.269172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923818752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077087.269177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077087.269179:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077087.269181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077087.269184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077087.269195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077087.269201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077087/real 1769077087] req@ffff969e0a7fc000 x1855007923818880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077087.269208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923818880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077087.269214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077087.269216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077087.269217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923818880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077087.588706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077088.292795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077088.612684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077089.316779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077089.636750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077090.340738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077090.660684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077091.364739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077091.684655:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077092.388698:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077092.388709:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077092.388713:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077092.388724:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3781 00000100:00080000:2.0:1769077092.388738:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077092.388745:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077092.388780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077092.388791:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923819008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077092.388836:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077092.388839:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077092.388841:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077092.388844:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3781 00000100:00080000:2.0:1769077092.388848:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077092.388852:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077092.388862:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077092.388870:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923819008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077092.388875:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923819136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077092.388884:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077092.388887:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077092.388888:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077092.388890:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3781 00000100:00080000:2.0:1769077092.388893:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077092.388896:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077092.388907:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077092.388922:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923819264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077092.388928:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077092.388930:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077092.388931:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077092.388935:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3781 00000100:00080000:2.0:1769077092.388944:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077092.388947:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077092.388951:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077092.388956:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077092.388966:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923819392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077092.388973:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077092.388976:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077092.388978:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077092.388981:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3781 00000100:00080000:2.0:1769077092.388999:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077092.389005:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077092.389009:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077092.389015:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077092/real 1769077092] req@ffff969e0a46e300 x1855007923819008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077092.389020:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923819520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077092.389048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923819008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077092.389058:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077092.389063:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077092.389066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923819008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077092.389092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077092.389117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077092.389131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077092/real 1769077092] req@ffff969e0a46ed80 x1855007923819136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077092.389145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923819136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077092.389153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077092.389156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077092.389158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077092.389163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077092.389178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077092.389185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077092/real 1769077092] req@ffff969e0a46ce00 x1855007923819264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077092.389193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923819264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077092.389209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077092.389211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077092.389214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077092.389218:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077092.389233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077092.389240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077092/real 1769077092] req@ffff969e0a46df80 x1855007923819392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077092.389247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923819392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077092.389253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077092.389255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077092.389257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077092.389260:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077092.389271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077092.389277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077092/real 1769077092] req@ffff969e0a46ca80 x1855007923819520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077092.389284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923819520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077092.389289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077092.389291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077092.389298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077092.708654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077093.412764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077093.732666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077094.436789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077094.756663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077095.460742:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077095.780693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077096.484771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077096.804685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077097.508728:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769077097.508729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077097.508738:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077097.508742:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077097.508750:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3786 00000100:00080000:0.0:1769077097.508759:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077097.508765:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077097.508795:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923819648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077097.508816:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077097.508819:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077097.508820:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077097.508823:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3786 00000100:00080000:0.0:1769077097.508826:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077097.508828:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077097.508836:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077097.508845:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923819776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077097.508852:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077097.508857:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769077097.508859:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923819648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077097.508873:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077097.508877:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3786 00000100:00080000:0.0:1769077097.508880:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077097.508883:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077097.508887:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077097.508898:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923819904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077097.508929:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077097.508931:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077097.508933:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077097.508936:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3786 00000100:00080000:0.0:1769077097.508960:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077097.508962:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077097.508966:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077097.508975:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923820032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077097.508981:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077097.508984:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077097.508985:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077097.508988:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3786 00000100:00080000:0.0:1769077097.508991:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077097.508994:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077097.508997:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077097.509008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077097.509014:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923820160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077097.509045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077097.509061:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077097.509076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077097.828722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077098.532803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077098.852689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077099.556751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077099.876700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077100.580907:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769077100.581041:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769077100.581103:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769077100.581133:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077100.581202:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077100.581214:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077100.581223:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077100.581232:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077100.581358:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077097/real 1769077100] req@ffff969e06a60a80 x1855007923820160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077100.581397:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923820160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077100.581439:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077100.581445:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077100.581450:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077100.581459:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077097/real 1769077100] req@ffff969e06a60000 x1855007923819776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077100.581474:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923819776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077100.581484:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077100.581489:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077100.581492:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077100.581499:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077097/real 1769077100] req@ffff969e06a63800 x1855007923819904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077100.581510:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923819904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077100.581517:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077100.581521:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077100.581525:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923819904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077100.581532:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077097/real 1769077100] req@ffff969e06a60700 x1855007923820032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077100.581544:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923820032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077100.581559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077100.581563:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077100.581567:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077100.581575:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077097/real 1769077100] req@ffff969e06a62d80 x1855007923819648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077100.581622:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923819648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077100.581633:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077100.581637:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077100.581640:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923819648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077100.900738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077101.604826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077101.924735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077102.628842:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077102.628861:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077102.628872:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077102.628878:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077102.628892:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3791 00000100:00080000:2.0:1769077102.628907:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077102.628915:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077102.628954:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923820288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077102.628987:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077102.629011:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077102.629015:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077102.629021:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3791 00000100:00080000:2.0:1769077102.629027:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077102.629032:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077102.629044:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077102.629049:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923820288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077102.629064:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923820416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077102.629081:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077102.629085:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077102.629087:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077102.629092:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3791 00000100:00080000:2.0:1769077102.629096:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077102.629101:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077102.629106:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077102.629119:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923820544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077102.629136:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077102.629147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077102.629150:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077102.629159:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077102.629168:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3791 00000100:00080000:2.0:1769077102.629177:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077102.629203:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077102.629226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077102/real 1769077102] req@ffff969e0a46ea00 x1855007923820288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077102.629249:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077102.629282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923820288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077102.629298:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923820672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077102.629309:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077102.629326:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077102.629328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769077102.629341:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077102.629345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923820288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077102.629348:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077102.629363:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3791 00000100:00080000:2.0:1769077102.629373:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077102.629386:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077102.629397:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077102.629417:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923820800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077102.629426:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077102.629489:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077102.629512:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077102/real 1769077102] req@ffff969e0a46f100 x1855007923820416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077102.629534:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923820416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077102.629548:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077102.629555:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077102.629561:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077102.629571:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077102.629601:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077102.629619:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077102/real 1769077102] req@ffff969e0a46d500 x1855007923820544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077102.629678:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923820544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077102.629692:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077102.629698:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077102.629703:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077102.629715:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077102.629752:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077102.629772:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077102/real 1769077102] req@ffff969e0a46f480 x1855007923820672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077102.629808:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923820672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077102.629847:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077102.629854:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077102.629859:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077102.629890:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077102.629929:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077102.629949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077102/real 1769077102] req@ffff969e0a46d880 x1855007923820800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077102.629970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923820800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077102.629983:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077102.629989:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077102.629995:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923820800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077102.948734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077103.653029:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077103.972675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077104.676756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077104.996671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077105.700732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077106.020691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077106.724821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077107.044661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077107.748718:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077107.748742:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077107.748747:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077107.748759:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3796 00000100:00080000:0.0:1769077107.748773:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077107.748781:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077107.748783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077107.748823:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923820928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077107.748848:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077107.748851:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077107.748852:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077107.748856:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3796 00000100:00080000:0.0:1769077107.748860:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077107.748863:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077107.748872:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077107.748882:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923820928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077107.748906:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923821056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077107.748915:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077107.748918:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077107.748920:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077107.748923:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3796 00000100:00080000:0.0:1769077107.748926:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077107.748929:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077107.748933:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077107.748940:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923821184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077107.748947:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077107.748950:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077107.748951:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077107.748954:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3796 00000100:00080000:0.0:1769077107.748957:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077107.748961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077107.748965:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077107.748970:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077107.748978:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923821312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077107.748985:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077107.748988:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077107.748990:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077107.748998:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3796 00000100:00100000:1.0:1769077107.749002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077107/real 1769077107] req@ffff969e06a61500 x1855007923820928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077107.749003:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077107.749007:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077107.749013:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077107.749028:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923821440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077107.749037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923820928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077107.749047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077107.749052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077107.749056:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923820928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077107.749079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077107.749096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077107.749105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077107/real 1769077107] req@ffff969e06a62680 x1855007923821056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077107.749114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923821056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077107.749120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077107.749122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077107.749124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077107.749128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077107.749138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077107.749143:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077107/real 1769077107] req@ffff969e06a61f80 x1855007923821184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077107.749151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923821184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077107.749166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077107.749169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077107.749171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077107.749175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077107.749189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077107.749195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077107/real 1769077107] req@ffff969e06a60380 x1855007923821312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077107.749203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923821312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077107.749208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077107.749210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077107.749212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077107.749215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077107.749225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077107.749231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077107/real 1769077107] req@ffff969e06a61180 x1855007923821440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077107.749238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923821440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077107.749243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077107.749245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077107.749252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077108.068719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077108.772817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077109.092826:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077109.796738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077110.116686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077110.820847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077111.140792:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077111.844772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077112.164705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077112.868692:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077112.868700:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077112.868720:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077112.868729:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3802 00010000:00010000:0.0:1769077112.868733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077112.868739:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077112.868746:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077112.868774:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923821568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077112.868796:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077112.868798:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077112.868800:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077112.868803:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3802 00000100:00080000:3.0:1769077112.868806:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077112.868809:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077112.868817:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077112.868830:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923821696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077112.868831:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923821568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077112.868845:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077112.868848:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077112.868849:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077112.868852:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3802 00000100:00080000:3.0:1769077112.868854:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077112.868857:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077112.868860:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077112.868869:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923821824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077112.868888:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077112.868896:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077112.868899:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077112.868902:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077112.868908:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3802 00000100:00080000:3.0:1769077112.868916:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077112.868922:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077112.868929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077112/real 1769077112] req@ffff969e0a7fd500 x1855007923821568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077112.868929:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077112.868951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923821568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077112.868959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923821952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077112.868960:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077112.868969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077112.868976:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077112.868977:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923821568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077112.868983:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077112.868986:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077112.868990:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3802 00000100:00080000:3.0:1769077112.868994:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077112.868996:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077112.869000:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077112.869004:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077112.869014:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923822080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077112.869024:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077112.869036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077112/real 1769077112] req@ffff969e0a7fc000 x1855007923821696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077112.869045:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923821696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077112.869051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077112.869052:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077112.869061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077112.869066:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077112.869080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077112.869086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077112/real 1769077112] req@ffff969e0a7ff800 x1855007923821824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077112.869094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923821824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077112.869099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077112.869101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077112.869103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077112.869107:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077112.869116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077112.869122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077112/real 1769077112] req@ffff969e0a7fdc00 x1855007923821952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077112.869129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923821952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077112.869134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077112.869136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077112.869138:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923821952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077112.869148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077112.869157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077112.869169:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077112/real 1769077112] req@ffff969e0a7ffb80 x1855007923822080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077112.869177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923822080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077112.869182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077112.869184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077112.869186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077113.188723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077113.892799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077114.212701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077114.916832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077115.236697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077115.940799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077116.260685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077116.964758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077117.284660:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077117.988747:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077117.988760:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077117.988766:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077117.988778:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3807 00010000:00010000:3.0:1769077117.988786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077117.988791:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077117.988802:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077117.988856:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923822208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077117.988891:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077117.988896:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077117.988898:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077117.988903:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3807 00000100:00080000:0.0:1769077117.988908:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077117.988913:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077117.988945:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077117.988952:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923822208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077117.988961:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923822336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077117.988971:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077117.988974:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077117.988975:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077117.988979:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3807 00000100:00080000:0.0:1769077117.988982:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077117.988985:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077117.988989:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077117.988999:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923822464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077117.989006:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077117.989009:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077117.989011:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077117.989013:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3807 00000100:00080000:0.0:1769077117.989025:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077117.989028:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077117.989032:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077117.989043:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923822592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077117.989044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077117.989054:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077117.989057:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077117.989062:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077117.989067:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3807 00000100:00080000:0.0:1769077117.989070:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077117.989074:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077117.989078:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077117.989087:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923822720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077117.989098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077117/real 1769077117] req@ffff969e06a61880 x1855007923822208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077117.989121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923822208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077117.989134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077117.989138:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077117.989141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923822208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077117.989164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077117.989185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077117.989206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077117/real 1769077117] req@ffff969e06a63100 x1855007923822336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077117.989229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923822336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077117.989239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077117.989242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077117.989246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077117.989253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077117.989276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077117.989287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077117/real 1769077117] req@ffff969e06a61180 x1855007923822464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077117.989297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923822464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077117.989303:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077117.989305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077117.989308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077117.989312:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077117.989338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077117.989345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077117/real 1769077117] req@ffff969e06a60380 x1855007923822592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077117.989352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923822592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077117.989368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077117.989372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077117.989376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077117.989382:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077117.989407:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077117.989424:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077117/real 1769077117] req@ffff969e06a61f80 x1855007923822720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077117.989434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923822720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077117.989441:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077117.989443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077117.989445:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077118.308679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077119.012782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077119.332697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077120.036757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077120.356670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077121.060813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077121.380678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077122.084824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077122.404684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077123.108716:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077123.108727:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077123.108732:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769077123.108734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077123.108743:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3812 00000100:00080000:3.0:1769077123.108771:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077123.108778:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077123.108831:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923822848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077123.108868:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077123.108873:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077123.108875:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077123.108880:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3812 00000100:00080000:3.0:1769077123.108884:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077123.108888:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077123.108897:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077123.108911:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923822976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077123.108922:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077123.108927:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077123.108929:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077123.108933:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3812 00000100:00080000:3.0:1769077123.108937:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077123.108940:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077123.108945:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077123.108964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923822848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077123.108983:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923823104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077123.108997:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077123.109001:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077123.109003:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077123.109007:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3812 00000100:00080000:3.0:1769077123.109013:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077123.109017:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077123.109023:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077123.109036:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923823232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077123.109047:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077123.109051:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077123.109052:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077123.109055:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3812 00000100:00080000:3.0:1769077123.109060:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077123.109068:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077123.109070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077123.109074:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077123.109087:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923823360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077123.109140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077123/real 1769077123] req@ffff969e0a7fd180 x1855007923822848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077123.109170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923822848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077123.109188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077123.109192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077123.109195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923822848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077123.109220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077123.109259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077123.109273:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077123/real 1769077123] req@ffff969e0a7fd880 x1855007923822976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077123.109303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923822976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077123.109311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077123.109315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077123.109318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923822976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077123.109340:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077123.109356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077123.109365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077123/real 1769077123] req@ffff969e0a7fca80 x1855007923823104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077123.109374:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923823104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077123.109382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077123.109384:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077123.109387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077123.109425:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077123.109447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077123.109457:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077123/real 1769077123] req@ffff969e0a7fe300 x1855007923823232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077123.109510:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923823232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077123.109537:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077123.109540:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077123.109543:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077123.109549:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077123.109569:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077123.109579:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077123/real 1769077123] req@ffff969e0a7fc380 x1855007923823360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077123.109589:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923823360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077123.109596:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077123.109599:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077123.109601:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077123.428669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077124.132764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077124.452671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077125.156815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077125.476726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077126.180978:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077126.500694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077127.205281:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077127.524699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077128.228865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077128.228933:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077128.228941:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077128.228947:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077128.228960:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3817 00000100:00080000:2.0:1769077128.228974:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077128.228981:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077128.229031:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923823488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077128.229062:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077128.229067:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077128.229069:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077128.229092:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3817 00000100:00080000:2.0:1769077128.229098:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077128.229103:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077128.229114:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077128.229118:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923823488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077128.229132:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923823616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077128.229151:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077128.229155:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077128.229157:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077128.229162:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3817 00000100:00080000:2.0:1769077128.229167:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077128.229171:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077128.229177:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077128.229191:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923823744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077128.229196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077128.229208:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077128.229212:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077128.229215:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077128.229219:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3817 00000100:00080000:2.0:1769077128.229223:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077128.229228:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077128.229233:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077128.229242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077128/real 1769077128] req@ffff969e0a46c380 x1855007923823488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077128.229247:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923823872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077128.229266:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077128.229271:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923823488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077128.229272:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077128.229277:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077128.229285:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3817 00000100:00080000:2.0:1769077128.229293:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077128.229295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077128.229299:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077128.229305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769077128.229307:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077128.229313:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923823488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077128.229322:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923824000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077128.229343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077128.229371:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077128.229418:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077128/real 1769077128] req@ffff969e0a46f800 x1855007923823616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077128.229434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923823616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077128.229443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077128.229446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077128.229450:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077128.229456:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077128.229479:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077128.229490:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077128/real 1769077128] req@ffff969e0a46d880 x1855007923823744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077128.229513:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923823744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077128.229528:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077128.229531:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077128.229535:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077128.229541:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077128.229562:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077128.229574:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077128/real 1769077128] req@ffff969e0a46f480 x1855007923823872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077128.229586:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923823872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077128.229624:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077128.229629:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077128.229634:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923823872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077128.229640:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077128.229660:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077128.229669:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077128/real 1769077128] req@ffff969e0a46d500 x1855007923824000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077128.229676:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923824000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077128.229692:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077128.229702:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077128.229707:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077128.548646:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077129.253652:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077129.572704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077130.276834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077130.597948:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077131.300815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077131.620676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077132.324762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077132.644745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077133.348718:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077133.348756:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077133.348761:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077133.348782:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3822 00000100:00080000:2.0:1769077133.348795:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077133.348811:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077133.348851:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923824128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769077133.348875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077133.348879:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077133.348885:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077133.348887:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077133.348893:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3822 00000100:00080000:2.0:1769077133.348900:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077133.348906:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077133.348920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923824128:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077133.348934:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077133.348953:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923824256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077133.348964:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077133.348967:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077133.348969:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077133.348974:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3822 00000100:00080000:2.0:1769077133.348978:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077133.348983:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077133.348987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769077133.348991:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077133.349006:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923824384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077133.349016:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077133.349021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077133/real 1769077133] req@ffff969e0a46f100 x1855007923824128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077133.349022:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077133.349026:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077133.349036:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3822 00000100:00080000:2.0:1769077133.349041:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077133.349046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923824128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077133.349047:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077133.349060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077133.349066:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769077133.349068:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077133.349072:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923824128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077133.349085:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923824512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077133.349095:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077133.349099:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077133.349107:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077133.349113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824256:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077133.349114:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3822 00000100:00080000:2.0:1769077133.349122:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077133.349128:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077133.349134:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077133.349145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077133.349151:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923824640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077133.349159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077133/real 1769077133] req@ffff969e0a46ea00 x1855007923824256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077133.349175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923824256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077133.349189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077133.349193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077133.349195:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077133.349200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077133.349214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077133.349221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077133/real 1769077133] req@ffff969e0a46ca80 x1855007923824384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077133.349228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923824384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077133.349233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077133.349235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077133.349237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077133.349241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077133.349251:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077133.349256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077133/real 1769077133] req@ffff969e0a46df80 x1855007923824512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077133.349264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923824512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077133.349269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077133.349270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077133.349277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077133.349291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077133.349320:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077133.349328:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077133/real 1769077133] req@ffff969e0a46ce00 x1855007923824640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077133.349335:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923824640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077133.349341:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077133.349343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077133.349345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077133.668710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077134.372803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077134.692698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077135.396746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077135.716717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077136.420734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077136.740689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077137.444796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077137.764775:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077138.468716:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077138.468729:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077138.468734:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077138.468744:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3827 00010000:00010000:2.0:1769077138.468757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077138.468757:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077138.468768:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077138.468811:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923824768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077138.468836:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077138.468839:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077138.468840:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077138.468844:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3827 00000100:00080000:3.0:1769077138.468847:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077138.468850:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077138.468860:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077138.468873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923824768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077138.468873:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923824896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077138.468890:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077138.468894:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077138.468896:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077138.468901:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3827 00000100:00080000:3.0:1769077138.468905:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077138.468933:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077138.468939:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077138.468949:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077138.468954:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923825024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077138.468962:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077138.468965:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077138.468968:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077138.468972:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3827 00000100:00080000:3.0:1769077138.468978:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077138.468982:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077138.468987:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077138/real 1769077138] req@ffff969e0a7fdf80 x1855007923824768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077138.468990:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077138.469008:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923824768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077138.469012:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923825152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077138.469024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077138.469031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077138.469031:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077138.469035:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077138.469036:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077138.469041:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3827 00000100:00100000:1.0:1769077138.469046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923824768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077138.469046:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077138.469053:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077138.469058:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077138.469068:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923825280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077138.469075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077138.469097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077138.469106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077138/real 1769077138] req@ffff969e0a7fea00 x1855007923824896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077138.469114:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923824896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077138.469120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077138.469122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077138.469124:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923824896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077138.469128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077138.469138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077138.469144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077138/real 1769077138] req@ffff969e0a7fc700 x1855007923825024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077138.469151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923825024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077138.469156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077138.469158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077138.469159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077138.469163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077138.469173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077138.469186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077138/real 1769077138] req@ffff969e0a7fce00 x1855007923825152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077138.469193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923825152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077138.469199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077138.469200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077138.469202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077138.469244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077138.469259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077138.469266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077138/real 1769077138] req@ffff969e0a7fe680 x1855007923825280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077138.469274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923825280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077138.469280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077138.469282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077138.469284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077138.788725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077139.492836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077139.812700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077140.516792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077140.836682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077141.540797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077141.860680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077142.564758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077142.884731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077143.588719:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077143.588735:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1769077143.588738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077143.588742:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077143.588756:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3832 00000100:00080000:2.0:1769077143.588771:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077143.588780:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077143.588830:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923825408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077143.588863:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077143.588868:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077143.588870:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077143.588877:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3832 00000100:00080000:2.0:1769077143.588883:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077143.588906:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923825408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077143.588914:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077143.588930:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077143.588951:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923825536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077143.588964:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077143.588968:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077143.588970:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077143.588975:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3832 00000800:00000100:1.0:1769077143.588980:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077143.588982:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077143.588989:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077143.588996:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077143.589017:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923825664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077143.589031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077143/real 1769077143] req@ffff969e0a46ed80 x1855007923825408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077143.589032:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077143.589048:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077143.589055:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077143.589066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923825408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077143.589066:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3832 00000100:00080000:2.0:1769077143.589080:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077143.589084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077143.589090:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077143.589093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077143.589102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923825408:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077143.589104:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077143.589123:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923825792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077143.589140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825536:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077143.589162:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077143.589170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077143.589180:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077143.589183:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077143.589188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077143/real 1769077143] req@ffff969e0a46e300 x1855007923825536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077143.589196:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3832 00000100:00100000:1.0:1769077143.589212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923825536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077143.589212:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077143.589228:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077143.589230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077143.589241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769077143.589244:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077143.589245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077143.589258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077143.589275:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923825920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077143.589289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077143.589312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077143/real 1769077143] req@ffff969e0a46ce00 x1855007923825664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077143.589329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923825664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077143.589345:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077143.589348:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077143.589358:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077143.589377:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077143.589394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077143.589402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077143/real 1769077143] req@ffff969e0a46df80 x1855007923825792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077143.589410:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923825792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077143.589415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077143.589418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077143.589420:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077143.589424:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077143.589434:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077143.589443:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077143/real 1769077143] req@ffff969e0a46ca80 x1855007923825920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077143.589455:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923825920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077143.589464:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077143.589467:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077143.589477:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923825920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077143.908677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077144.612814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077144.932683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077145.636792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077145.956669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077146.660798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077146.980688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077147.684728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077148.004764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077148.708751:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769077148.708757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077148.708761:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077148.708769:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077148.708779:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3837 00000100:00080000:0.0:1769077148.708790:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077148.708796:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077148.708832:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923826048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077148.708857:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077148.708860:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077148.708861:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077148.708865:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3837 00000100:00080000:0.0:1769077148.708867:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077148.708871:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077148.708879:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077148.708904:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923826048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077148.708908:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923826176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077148.708928:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077148.708930:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077148.708932:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077148.708936:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3837 00000100:00080000:0.0:1769077148.708942:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077148.708945:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077148.708949:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077148.708957:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923826304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077148.708964:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077148.708968:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077148.708970:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077148.708972:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3837 00000100:00080000:0.0:1769077148.708981:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077148.708985:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077148.708989:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077148.708995:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077148.709002:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923826432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077148.709011:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077148.709014:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077148.709016:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077148.709025:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3837 00000100:00080000:0.0:1769077148.709031:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077148.709036:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077148.709055:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077148.709060:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077148/real 1769077148] req@ffff969e06a62680 x1855007923826048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077148.709070:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923826560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077148.709096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923826048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077148.709114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077148.709121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077148.709125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923826048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077148.709156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077148.709177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077148.709188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077148/real 1769077148] req@ffff969e06a61500 x1855007923826176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077148.709197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923826176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077148.709203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077148.709206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077148.709219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077148.709224:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077148.709240:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077148.709247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077148/real 1769077148] req@ffff969e06a62d80 x1855007923826304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077148.709255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923826304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077148.709261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077148.709263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077148.709265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077148.709269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077148.709281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077148.709287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077148/real 1769077148] req@ffff969e06a60700 x1855007923826432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077148.709295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923826432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077148.709300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077148.709302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077148.709304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077148.709308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077148.709317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077148.709330:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077148/real 1769077148] req@ffff969e06a63800 x1855007923826560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077148.709338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923826560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077148.709347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077148.709361:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077148.709365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077149.028737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077149.732759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077150.052680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077150.756723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077151.076722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077151.780937:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077152.100684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077152.804785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077153.124701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077153.828757:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077153.828768:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077153.828772:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077153.828782:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3843 00000100:00080000:0.0:1769077153.828806:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077153.828814:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077153.828836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077153.828861:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923826688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077153.828891:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077153.828894:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077153.828896:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077153.828900:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3843 00000100:00080000:0.0:1769077153.828903:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077153.828907:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077153.828917:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077153.828942:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923826816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077153.828950:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077153.828979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923826688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077153.828982:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077153.829002:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077153.829015:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3843 00000100:00080000:0.0:1769077153.829020:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077153.829027:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077153.829033:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077153.829051:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923826944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077153.829063:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077153.829066:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077153.829068:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077153.829072:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3843 00000100:00080000:0.0:1769077153.829076:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077153.829079:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077153.829082:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077153.829090:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077153.829101:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923827072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077153.829110:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077153.829113:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077153.829115:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077153.829120:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3843 00000100:00080000:0.0:1769077153.829126:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077153.829129:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077153.829136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077153/real 1769077153] req@ffff969e06a60000 x1855007923826688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077153.829141:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077153.829163:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923827200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077153.829167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923826688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077153.829205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077153.829210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077153.829213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923826688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077153.829241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077153.829258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077153.829281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077153/real 1769077153] req@ffff969e06a60a80 x1855007923826816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077153.829289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923826816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077153.829296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077153.829299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077153.829301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077153.829306:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077153.829319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077153.829325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077153/real 1769077153] req@ffff969e06a63800 x1855007923826944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077153.829341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923826944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077153.829347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077153.829350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077153.829352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923826944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077153.829356:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077153.829368:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077153.829374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077153/real 1769077153] req@ffff969e06a60700 x1855007923827072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077153.829382:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923827072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077153.829393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077153.829396:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077153.829398:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077153.829402:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077153.829416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077153.829423:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077153/real 1769077153] req@ffff969e06a62d80 x1855007923827200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077153.829430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923827200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077153.829436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077153.829438:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077153.829439:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077154.148727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077154.852802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077155.172705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077155.876849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077156.196800:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077156.900879:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077157.220723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077157.924864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077158.244739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077158.948694:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077158.948735:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077158.948741:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077158.948756:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3848 00000100:00080000:3.0:1769077158.948771:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077158.948781:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077158.948807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077158.948853:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923827328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077158.948910:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077158.948913:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077158.948914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077158.948936:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3848 00000100:00080000:3.0:1769077158.948940:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077158.948944:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077158.948949:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923827328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077158.948956:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077158.948973:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923827456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077158.948981:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077158.948984:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077158.948985:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077158.948988:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3848 00000100:00080000:3.0:1769077158.948991:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077158.948993:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077158.948996:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077158.949003:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923827584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077158.949010:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077158.949012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077158.949014:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077158.949017:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3848 00000100:00080000:3.0:1769077158.949020:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077158.949024:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077158.949026:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077158.949031:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077158.949039:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923827712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077158.949051:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077158.949054:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077158.949056:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077158.949059:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3848 00000100:00100000:1.0:1769077158.949062:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077158/real 1769077158] req@ffff969e0a7fed80 x1855007923827328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077158.949064:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077158.949072:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077158.949079:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077158.949086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923827328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077158.949090:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923827840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077158.949117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077158.949122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077158.949126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923827328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077158.949149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077158.949167:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077158.949175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077158/real 1769077158] req@ffff969e0a7fe680 x1855007923827456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077158.949184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923827456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077158.949190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077158.949192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077158.949194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077158.949198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077158.949208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077158.949213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077158/real 1769077158] req@ffff969e0a7fce00 x1855007923827584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077158.949220:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923827584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077158.949226:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077158.949228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077158.949229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077158.949239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077158.949252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077158.949259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077158/real 1769077158] req@ffff969e0a7fc700 x1855007923827712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077158.949266:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923827712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077158.949272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077158.949273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077158.949275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077158.949279:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077158.949289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077158.949294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077158/real 1769077158] req@ffff969e0a7fea00 x1855007923827840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077158.949301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923827840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077158.949306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077158.949308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077158.949310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923827840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077159.268682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077159.972831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077160.292663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077160.996806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077161.316691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077162.020804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077162.340679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077163.044756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077163.364680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077164.068724:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077164.068766:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077164.068773:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077164.068777:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077164.068817:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3853 00000100:00080000:0.0:1769077164.068830:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077164.068837:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077164.068878:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923827968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077164.068903:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077164.068906:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077164.068907:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077164.068911:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3853 00000100:00080000:0.0:1769077164.068915:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077164.068919:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077164.068927:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077164.068935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923827968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077164.068940:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923828096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077164.068949:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077164.068952:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077164.068953:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077164.068956:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3853 00000100:00080000:0.0:1769077164.068974:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077164.068978:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077164.068982:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077164.068990:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923828224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077164.068997:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077164.068999:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077164.069001:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077164.069003:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3853 00000100:00080000:0.0:1769077164.069006:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077164.069008:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077164.069011:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077164.069019:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923828352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077164.069025:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077164.069027:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077164.069028:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077164.069031:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3853 00000100:00080000:0.0:1769077164.069034:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077164.069036:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828096:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769077164.069039:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077164.069045:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077164.069055:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923828480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077164.069066:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077164.069092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077164.069111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077164.388679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077165.092791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077165.412682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077166.116810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077166.436675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077167.140839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077167.140942:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769077167.141010:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077167.141046:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077167.141097:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077167.141108:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077167.141116:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077167.141126:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077167.141287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077164/real 1769077167] req@ffff969e06a61180 x1855007923828480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077167.141333:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923828480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077167.141350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077167.141357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077167.141363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077167.141374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077164/real 1769077167] req@ffff969e06a62680 x1855007923828096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077167.141389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923828096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077167.141399:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077167.141427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077167.141431:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077167.141447:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077164/real 1769077167] req@ffff969e06a61f80 x1855007923828224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077167.141465:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923828224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077167.141474:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077167.141479:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077167.141482:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077167.141491:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077164/real 1769077167] req@ffff969e06a60380 x1855007923828352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077167.141502:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923828352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077167.141511:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077167.141515:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077167.141519:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077167.141527:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077164/real 1769077167] req@ffff969e06a61500 x1855007923827968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077167.141541:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923827968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077167.141550:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077167.141563:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077167.141569:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923827968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077167.460673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077168.164900:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077168.484741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077169.188736:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077169.188751:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077169.188757:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077169.188781:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3858 00010000:00010000:3.0:1769077169.188784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077169.188793:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077169.188803:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077169.188850:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923828608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077169.188886:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077169.188890:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077169.188892:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077169.188897:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3858 00000100:00080000:0.0:1769077169.188902:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077169.188907:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077169.188928:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077169.188933:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923828608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077169.188958:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923828736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077169.188973:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077169.188977:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077169.188980:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077169.188986:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3858 00000100:00080000:0.0:1769077169.188991:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077169.188996:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077169.189033:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077169.189055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077169.189056:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923828864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077169.189070:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077169.189075:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077169.189088:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077169.189093:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3858 00000100:00080000:0.0:1769077169.189099:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077169.189104:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077169.189111:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077169.189124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077169/real 1769077169] req@ffff969e06a63100 x1855007923828608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077169.189126:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923828992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077169.189163:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077169.189168:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077169.189170:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077169.189177:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3858 00000100:00100000:1.0:1769077169.189179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923828608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077169.189183:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077169.189193:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077169.189204:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077169.189205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077169.189219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077169.189221:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923829120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077169.189233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923828608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077169.189271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077169.189312:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077169.189329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077169/real 1769077169] req@ffff969e06a61880 x1855007923828736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077169.189345:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923828736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077169.189356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077169.189359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077169.189362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077169.189389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077169.189417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077169.189431:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077169/real 1769077169] req@ffff969e06a61500 x1855007923828864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077169.189445:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923828864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077169.189455:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077169.189459:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077169.189464:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077169.189472:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077169.189496:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077169.189522:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077169/real 1769077169] req@ffff969e06a60380 x1855007923828992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077169.189536:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923828992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077169.189546:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077169.189550:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077169.189554:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923828992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077169.189561:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077169.189585:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077169.189628:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077169/real 1769077169] req@ffff969e06a61f80 x1855007923829120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077169.189653:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923829120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077169.189665:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077169.189669:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077169.189673:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077169.508661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077170.213927:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077170.532722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077171.236789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077171.556716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077172.260805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077172.580718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077173.284790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077173.604734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077174.308800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077174.308801:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077174.308812:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077174.308816:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077174.308826:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3863 00000100:00080000:2.0:1769077174.308836:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077174.308842:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077174.308878:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923829248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077174.308901:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077174.308919:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077174.308921:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077174.308926:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3863 00000100:00080000:2.0:1769077174.308930:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077174.308933:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077174.308942:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077174.308943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923829248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077174.308955:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923829376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077174.308967:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077174.308971:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077174.308973:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077174.308976:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3863 00000100:00080000:2.0:1769077174.308989:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077174.308993:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077174.308996:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077174.309008:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923829504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077174.309014:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077174.309017:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077174.309019:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077174.309022:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3863 00000100:00080000:2.0:1769077174.309032:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077174.309034:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077174.309045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769077174.309058:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077174.309074:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923829632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077174.309083:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077174.309087:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077174.309088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077174/real 1769077174] req@ffff969e0a46ea00 x1855007923829248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077174.309091:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077174.309101:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3863 00000100:00080000:2.0:1769077174.309112:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077174.309119:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077174.309125:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077174.309129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923829248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077174.309137:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923829760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077174.309152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077174.309161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077174.309167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923829248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077174.309203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077174.309222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077174.309231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077174/real 1769077174] req@ffff969e0a46f100 x1855007923829376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077174.309252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923829376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077174.309259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077174.309278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077174.309281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077174.309286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077174.309301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077174.309308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077174/real 1769077174] req@ffff969e0a46d500 x1855007923829504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077174.309324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923829504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077174.309330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077174.309333:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077174.309335:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077174.309340:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077174.309352:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077174.309358:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077174/real 1769077174] req@ffff969e0a46f480 x1855007923829632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077174.309366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923829632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077174.309382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077174.309385:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077174.309387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077174.309392:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077174.309406:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077174.309412:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077174/real 1769077174] req@ffff969e0a46d880 x1855007923829760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077174.309419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923829760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077174.309425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077174.309427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077174.309429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923829760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077174.628700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077175.332862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077175.652780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077176.356829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077176.676683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077177.380799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077177.700701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077178.404811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077178.724682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077179.428769:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077179.428780:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077179.428784:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077179.428794:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3868 00000100:00080000:3.0:1769077179.428825:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077179.428836:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077179.428853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077179.428894:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923829888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077179.428946:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077179.428966:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077179.428968:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077179.428972:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3868 00000100:00080000:3.0:1769077179.428977:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077179.428982:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077179.428992:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077179.429032:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923830016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077179.429040:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077179.429043:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077179.429045:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077179.429048:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3868 00000100:00080000:3.0:1769077179.429051:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077179.429054:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077179.429076:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077179.429079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923829888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077179.429092:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923830144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077179.429120:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077179.429125:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077179.429126:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077179.429129:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3868 00000100:00080000:3.0:1769077179.429131:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077179.429134:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077179.429137:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077179.429148:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923830272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077179.429157:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077179.429159:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077179.429160:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077179.429164:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3868 00000100:00080000:3.0:1769077179.429167:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077179.429169:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077179.429172:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077179.429179:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923830400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077179.429260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077179.429340:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077179/real 1769077179] req@ffff969e0a7fdf80 x1855007923829888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077179.429381:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923829888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077179.429438:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077179.429450:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077179.429461:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923829888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077179.429502:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077179.429654:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077179.429704:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077179/real 1769077179] req@ffff969e0a7fc380 x1855007923830016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077179.429744:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923830016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077179.429763:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077179.429771:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077179.429779:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077179.429845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077179.429919:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077179.429940:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077179/real 1769077179] req@ffff969e0a7fe300 x1855007923830144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077179.429972:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923830144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077179.430009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077179.430026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077179.430035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077179.430065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077179.430135:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077179.430192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077179/real 1769077179] req@ffff969e0a7fca80 x1855007923830272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077179.430241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923830272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077179.430260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077179.430268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077179.430275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077179.430288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077179.430336:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077179.430359:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077179/real 1769077179] req@ffff969e0a7fd880 x1855007923830400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077179.430384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923830400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077179.430409:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077179.430417:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077179.430424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077179.748697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077180.452776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077180.772704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077181.476810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077181.796722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077182.500740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077182.820723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077183.524777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077183.844675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077184.548661:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077184.548673:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077184.548678:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077184.548687:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3873 00000100:00080000:3.0:1769077184.548699:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077184.548707:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077184.548730:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077184.548752:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923830528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077184.548776:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077184.548779:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077184.548781:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077184.548784:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3873 00000100:00080000:3.0:1769077184.548786:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077184.548789:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077184.548798:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077184.548807:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923830656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077184.548816:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077184.548818:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077184.548819:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077184.548825:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923830528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077184.548825:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3873 00000100:00080000:3.0:1769077184.548831:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077184.548835:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077184.548858:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077184.548868:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923830784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077184.548876:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077184.548879:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077184.548880:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077184.548884:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3873 00000100:00080000:3.0:1769077184.548892:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077184.548897:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077184.548902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077184.548904:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077184.548923:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923830912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077184.548934:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077184.548939:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077184/real 1769077184] req@ffff969e0a7fd180 x1855007923830528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077184.548940:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077184.548944:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077184.548954:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3873 00000100:00080000:3.0:1769077184.548960:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077184.548965:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923830528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077184.548966:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077184.548976:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077184.548980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077184.548986:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077184.548992:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923830528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077184.548993:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923831040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077184.549018:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077184.549045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077184.549054:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077184/real 1769077184] req@ffff969e0a7ffb80 x1855007923830656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077184.549062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923830656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077184.549069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077184.549071:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077184.549073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077184.549078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077184.549089:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077184.549094:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077184/real 1769077184] req@ffff969e0a7fdc00 x1855007923830784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077184.549110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923830784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077184.549116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077184.549119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077184.549121:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077184.549124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077184.549138:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077184.549144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077184/real 1769077184] req@ffff969e0a7ff800 x1855007923830912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077184.549152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923830912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077184.549158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077184.549159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077184.549161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923830912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077184.549164:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077184.549174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077184.549180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077184/real 1769077184] req@ffff969e0a7fc000 x1855007923831040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077184.549187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923831040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077184.549202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077184.549204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077184.549207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077184.868673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077185.572772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077185.892733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077186.596770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077186.916669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077187.620799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077187.940744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077188.644741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077188.964708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077189.668705:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077189.668726:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077189.668748:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077189.668761:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3878 00010000:00010000:2.0:1769077189.668763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077189.668774:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077189.668783:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077189.668819:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923831168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077189.668850:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077189.668855:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077189.668857:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077189.668863:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3878 00000100:00080000:3.0:1769077189.668868:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077189.668874:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077189.668880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923831168:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077189.668887:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077189.668907:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923831296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077189.668920:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077189.668925:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077189.668927:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077189.668931:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3878 00000100:00080000:3.0:1769077189.668936:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077189.668942:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077189.668943:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077189.668951:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077189.668964:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923831424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077189.668975:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077189/real 1769077189] req@ffff969e0a7fd500 x1855007923831168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077189.669002:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077189.669003:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923831168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077189.669012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077189.669017:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077189.669018:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077189.669025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077189.669025:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3878 00000100:00100000:1.0:1769077189.669032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923831168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077189.669033:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077189.669042:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077189.669048:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077189.669055:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077189.669064:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923831552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077189.669075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077189.669078:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077189.669084:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077189.669087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077189/real 1769077189] req@ffff969e0a7fc000 x1855007923831296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077189.669087:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077189.669097:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3878 00000100:00100000:1.0:1769077189.669103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923831296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077189.669104:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077189.669111:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077189.669120:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077189.669122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077189.669128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077189.669131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077189.669137:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923831680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077189.669139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077189.669163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077189.669173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077189/real 1769077189] req@ffff969e0a7ff800 x1855007923831424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077189.669181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923831424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077189.669196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077189.669198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077189.669200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077189.669211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077189.669224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077189.669230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077189/real 1769077189] req@ffff969e0a7fdc00 x1855007923831552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077189.669238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923831552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077189.669243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077189.669246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077189.669252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077189.669257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077189.669270:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077189.669276:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077189/real 1769077189] req@ffff969e0a7ffb80 x1855007923831680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077189.669283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923831680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077189.669290:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077189.669292:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077189.669295:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077189.988711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077190.692762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077191.012715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077191.717542:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077192.036758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077192.741587:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077193.060715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077193.764892:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077194.084731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077194.788973:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077194.788987:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077194.788992:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077194.789003:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3883 00000100:00080000:2.0:1769077194.789017:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077194.789026:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077194.789088:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923831808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077194.789126:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077194.789131:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077194.789134:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077194.789141:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3883 00000100:00080000:2.0:1769077194.789147:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077194.789151:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077194.789164:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077194.789182:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923831936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077194.789195:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077194.789199:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077194.789202:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077194.789207:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3883 00000100:00080000:2.0:1769077194.789214:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077194.789221:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077194.789230:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077194.789237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923831808:192.168.204.143@tcp:250:kworker.0 00010000:00010000:3.0:1769077194.789242:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077194.789260:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923832064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077194.789297:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077194.789301:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077194.789303:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077194.789318:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3883 00000100:00080000:2.0:1769077194.789325:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077194.789330:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077194.789338:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077194.789352:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923832192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077194.789364:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077194.789367:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077194.789370:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077194.789378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077194.789403:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3883 00000100:00080000:2.0:1769077194.789411:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077194.789417:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077194.789424:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077194.789430:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077194/real 1769077194] req@ffff969e0a46f800 x1855007923831808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077194.789448:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923832320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077194.789460:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923831808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077194.789487:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077194.789493:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077194.789512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923831808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077194.789524:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077194.789561:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077194.789576:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077194/real 1769077194] req@ffff969e0a46c380 x1855007923831936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077194.789620:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923831936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077194.789637:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077194.789642:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077194.789646:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923831936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077194.789678:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077194.789699:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077194.789712:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077194/real 1769077194] req@ffff969e0a46d880 x1855007923832064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077194.789725:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923832064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077194.789733:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077194.789737:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077194.789745:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077194.789753:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077194.789777:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077194.789799:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077194/real 1769077194] req@ffff969e0a46f480 x1855007923832192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077194.789812:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923832192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077194.789819:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077194.789822:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077194.789824:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077194.789829:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077194.789845:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077194.789853:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077194/real 1769077194] req@ffff969e0a46d500 x1855007923832320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077194.789861:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923832320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077194.789867:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077194.789868:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077194.789871:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077195.108735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077195.812991:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077196.132684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077196.837396:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077197.156683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077197.860791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077198.180711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077198.884792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077199.204701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077199.908740:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077199.908750:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077199.908782:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077199.908792:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3889 00010000:00010000:3.0:1769077199.908794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077199.908804:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077199.908811:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077199.908851:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923832448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077199.908875:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077199.908878:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077199.908879:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077199.908883:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3889 00000100:00080000:0.0:1769077199.908885:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077199.908888:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077199.908896:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077199.908918:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923832576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077199.908926:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077199.908929:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077199.908931:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077199.908937:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3889 00000100:00100000:1.0:1769077199.908937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923832448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077199.908941:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077199.908948:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077199.908953:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077199.908962:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923832704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077199.908979:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077199.908982:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077199.908993:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077199.908996:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3889 00000100:00080000:0.0:1769077199.908999:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077199.909003:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077199.909006:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077199.909015:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923832832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077199.909021:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077199.909024:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077199.909025:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077199.909027:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3889 00000100:00080000:0.0:1769077199.909029:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077199.909035:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077199.909036:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077199.909041:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077199.909054:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923832960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077199.909083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077199/real 1769077199] req@ffff969e06a62680 x1855007923832448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077199.909100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923832448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077199.909108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077199.909112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077199.909115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923832448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077199.909146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077199.909165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077199.909174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077199/real 1769077199] req@ffff969e06a61180 x1855007923832576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077199.909213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923832576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077199.909220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077199.909223:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077199.909225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077199.909230:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077199.909246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077199.909254:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077199/real 1769077199] req@ffff969e06a62d80 x1855007923832704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077199.909275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923832704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077199.909291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077199.909294:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077199.909296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077199.909301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077199.909316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077199.909323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077199/real 1769077199] req@ffff969e06a60700 x1855007923832832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077199.909332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923832832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077199.909338:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077199.909340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077199.909342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077199.909346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077199.909362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077199.909373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077199/real 1769077199] req@ffff969e06a63800 x1855007923832960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077199.909385:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923832960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077199.909395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077199.909399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077199.909410:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923832960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077200.228679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077200.932795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077201.252713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077201.956762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077202.276713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077202.980871:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077203.300680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077204.004720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077204.324744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077205.028728:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077205.028740:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077205.028760:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077205.028772:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3894 00000100:00080000:0.0:1769077205.028802:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077205.028810:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077205.028846:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923833088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077205.028895:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077205.028898:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077205.028901:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077205.028906:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3894 00000100:00080000:0.0:1769077205.028911:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077205.028916:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077205.028929:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:0.0:1769077205.028932:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077205.028946:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923833216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077205.028968:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923833088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077205.028981:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077205.028987:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077205.028989:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077205.028996:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3894 00000100:00080000:0.0:1769077205.029002:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077205.029007:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077205.029013:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077205.029028:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923833344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077205.029040:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077205.029052:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1769077205.029057:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077205.029059:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077205.029068:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3894 00000100:00080000:0.0:1769077205.029108:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077205.029113:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077205.029121:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077205.029135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077205/real 1769077205] req@ffff969e06a60a80 x1855007923833088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077205.029140:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923833472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077205.029182:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077205.029185:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077205.029188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923833088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077205.029190:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077205.029207:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3894 00000100:00080000:1.0:1769077205.029215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077205.029216:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077205.029226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769077205.029230:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077205.029237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923833088:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077205.029242:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077205.029258:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923833600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077205.029267:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077205.029290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077205.029303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077205/real 1769077205] req@ffff969e06a60000 x1855007923833216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077205.029311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923833216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077205.029317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077205.029320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077205.029335:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077205.029340:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077205.029355:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077205.029362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077205/real 1769077205] req@ffff969e06a63800 x1855007923833344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077205.029369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923833344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077205.029375:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077205.029377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077205.029379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077205.029383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077205.029394:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077205.029399:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077205/real 1769077205] req@ffff969e06a60700 x1855007923833472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077205.029407:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923833472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077205.029412:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077205.029414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077205.029416:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077205.029427:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077205.029444:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077205.029465:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077205/real 1769077205] req@ffff969e06a62d80 x1855007923833600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077205.029478:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923833600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077205.029487:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077205.029491:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077205.029495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077205.348699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077206.052762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077206.372723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077207.076758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077207.396718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077208.100776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077208.420709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077209.124791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077209.444681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077210.148718:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077210.148732:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077210.148737:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077210.148747:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3899 00000100:00080000:2.0:1769077210.148759:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077210.148766:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077210.148789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077210.148810:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923833728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077210.148833:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077210.148853:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077210.148856:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077210.148859:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3899 00000100:00080000:2.0:1769077210.148862:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077210.148867:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077210.148871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923833728:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077210.148878:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077210.148891:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923833856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077210.148900:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077210.148902:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077210.148904:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077210.148907:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3899 00000100:00080000:2.0:1769077210.148910:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077210.148912:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077210.148915:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077210.148923:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923833984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077210.148930:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077210.148933:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077210.148935:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077210.148937:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3899 00000100:00080000:2.0:1769077210.148939:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077210.148943:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077210.148945:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769077210.148954:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077210.148964:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923834112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077210.148972:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077210.148975:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077210.148976:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077210.148979:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3899 00000100:00100000:1.0:1769077210.148982:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077210/real 1769077210] req@ffff969e0a46f100 x1855007923833728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077210.148984:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077210.148989:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077210.148992:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077210.149002:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923834240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077210.149026:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923833728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077210.149055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077210.149061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077210.149065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923833728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077210.149091:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077210.149118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077210.149147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077210/real 1769077210] req@ffff969e0a46ea00 x1855007923833856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077210.149165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923833856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077210.149194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077210.149199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077210.149204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077210.149212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077210.149234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077210.149244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077210/real 1769077210] req@ffff969e0a46ca80 x1855007923833984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077210.149255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923833984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077210.149263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077210.149266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077210.149269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923833984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077210.149274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077210.149287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077210.149295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077210/real 1769077210] req@ffff969e0a46df80 x1855007923834112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077210.149304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923834112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077210.149320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077210.149323:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077210.149326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077210.149331:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077210.149348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077210.149356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077210/real 1769077210] req@ffff969e0a46ce00 x1855007923834240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077210.149366:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923834240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077210.149374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077210.149376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077210.149379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077210.468713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077211.172763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077211.492734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077212.196933:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077212.516680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077213.220739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077213.540713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077214.244807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077214.564681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077215.268758:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077215.268786:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077215.268792:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077215.268805:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3904 00010000:00010000:3.0:1769077215.268817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077215.268818:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077215.268829:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077215.268879:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923834368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077215.268981:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077215.268987:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077215.268991:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077215.268999:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3904 00000100:00080000:0.0:1769077215.269006:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077215.269018:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077215.269030:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077215.269032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923834368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077215.269055:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923834496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077215.269087:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077215.269091:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077215.269095:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077215.269101:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3904 00000100:00080000:0.0:1769077215.269118:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077215.269123:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077215.269128:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077215.269136:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077215.269141:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923834624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077215.269153:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077215.269157:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077215.269159:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077215.269166:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3904 00000100:00080000:0.0:1769077215.269178:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077215.269182:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077215.269189:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077215.269193:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077215/real 1769077215] req@ffff969e06a61180 x1855007923834368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077215.269203:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923834752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077215.269256:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077215.269260:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077215.269262:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077215.269269:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3904 00000100:00080000:0.0:1769077215.269272:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077215.269279:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923834368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077215.269280:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077215.269284:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077215.269291:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923834880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077215.269320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077215.269331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077215.269337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923834368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077215.269365:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077215.269395:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077215.269411:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077215/real 1769077215] req@ffff969e06a62680 x1855007923834496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077215.269434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923834496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077215.269443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077215.269447:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077215.269451:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077215.269458:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077215.269475:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077215.269484:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077215/real 1769077215] req@ffff969e06a61f80 x1855007923834624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077215.269491:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923834624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077215.269497:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077215.269517:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077215.269520:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077215.269540:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077215.269574:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077215.269591:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077215/real 1769077215] req@ffff969e06a60380 x1855007923834752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077215.269610:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923834752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077215.269657:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077215.269663:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077215.269669:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077215.269680:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077215.269712:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077215.269729:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077215/real 1769077215] req@ffff969e06a61500 x1855007923834880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077215.269747:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923834880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077215.269760:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077215.269764:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077215.269767:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923834880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077215.588750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077216.292780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077216.612683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077217.316778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077217.636690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077218.340861:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077218.660707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077219.364819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077219.684682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077220.388774:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769077220.388790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077220.388791:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077220.388800:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077220.388814:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3909 00000100:00080000:0.0:1769077220.388829:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077220.388874:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077220.388941:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923835008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077220.388977:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077220.389005:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077220.389009:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077220.389016:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3909 00000100:00080000:0.0:1769077220.389034:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077220.389041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923835008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077220.389044:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077220.389060:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077220.389081:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923835136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077220.389108:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077220.389112:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077220.389115:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077220.389121:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3909 00000100:00080000:0.0:1769077220.389127:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077220.389133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077220.389138:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077220.389146:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077220.389166:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923835264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077220.389176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077220/real 1769077220] req@ffff969e06a61880 x1855007923835008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077220.389185:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077220.389193:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077220.389196:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077220.389199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923835008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077220.389206:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3909 00000100:00080000:0.0:1769077220.389212:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077220.389219:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077220.389239:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077220.389242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077220.389249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077220.389254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923835008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077220.389262:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923835392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077220.389277:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077220.389283:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077220.389286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835136:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769077220.389288:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077220.389299:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3909 00000100:00080000:0.0:1769077220.389316:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077220.389323:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077220.389326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769077220.389335:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077220.389342:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077220/real 1769077220] req@ffff969e06a63100 x1855007923835136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077220.389353:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923835520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077220.389356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923835136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077220.389391:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077220.389397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077220.389401:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077220.389407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077220.389434:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077220.389445:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077220/real 1769077220] req@ffff969e06a61500 x1855007923835264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077220.389471:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923835264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077220.389482:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077220.389486:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077220.389491:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077220.389500:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077220.389529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077220.389539:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077220/real 1769077220] req@ffff969e06a60380 x1855007923835392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077220.389549:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923835392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077220.389556:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077220.389558:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077220.389561:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077220.389574:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077220.389631:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077220.389640:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077220/real 1769077220] req@ffff969e06a61f80 x1855007923835520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077220.389649:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923835520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077220.389671:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077220.389676:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077220.389680:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077220.708729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077221.412812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077221.732711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077222.436841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077222.756772:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077223.460819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077223.780673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077224.484748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077224.804724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077225.508733:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077225.508758:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077225.508764:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077225.508779:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3914 00000100:00080000:0.0:1769077225.508794:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077225.508822:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077225.508834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077225.508881:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923835648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077225.508904:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077225.508907:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077225.508909:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077225.508912:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3914 00000100:00080000:0.0:1769077225.508933:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077225.508937:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077225.508947:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077225.508955:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923835648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077225.508961:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923835776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077225.508971:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077225.508974:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077225.508977:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077225.508982:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3914 00000100:00080000:0.0:1769077225.508986:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077225.508990:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077225.508996:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077225.509009:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923835904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077225.509021:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077225.509028:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077225.509029:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077225.509034:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077225.509040:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3914 00000100:00080000:0.0:1769077225.509047:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077225.509052:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077225.509059:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077225.509063:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077225/real 1769077225] req@ffff969e06a62680 x1855007923835648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077225.509078:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923836032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077225.509081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923835648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077225.509110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077225.509117:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077225.509125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923835648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077225.509131:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077225.509136:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077225.509139:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077225.509145:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3914 00000100:00080000:0.0:1769077225.509152:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077225.509152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835776:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769077225.509164:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077225.509171:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077225.509176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077225.509187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077225/real 1769077225] req@ffff969e06a61180 x1855007923835776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077225.509193:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923836160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077225.509200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923835776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077225.509215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077225.509219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077225.509222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077225.509228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077225.509245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077225.509255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077225/real 1769077225] req@ffff969e06a62d80 x1855007923835904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077225.509264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923835904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077225.509271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077225.509274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077225.509276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923835904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077225.509279:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077225.509289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077225.509295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077225/real 1769077225] req@ffff969e06a60700 x1855007923836032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077225.509302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923836032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077225.509308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077225.509309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077225.509316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077225.509328:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077225.509342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077225.509348:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077225/real 1769077225] req@ffff969e06a63800 x1855007923836160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077225.509356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923836160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077225.509362:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077225.509364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077225.509365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077225.828669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077226.532774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077226.852723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077227.556756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077227.876737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077228.580805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077228.900754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077229.604832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077229.924724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077230.628775:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077230.628784:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077230.628788:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077230.628797:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3919 00000100:00080000:3.0:1769077230.628808:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077230.628814:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077230.628847:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923836288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769077230.628850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077230.628876:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077230.628879:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077230.628881:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077230.628884:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3919 00000100:00080000:3.0:1769077230.628887:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077230.628891:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077230.628898:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077230.628908:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923836416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077230.628917:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077230.628919:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077230.628920:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077230.628925:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3919 00000100:00100000:1.0:1769077230.628926:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923836288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077230.628927:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077230.628933:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077230.628951:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077230.628962:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923836544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077230.628969:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077230.628972:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077230.628973:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077230.628976:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3919 00000100:00080000:3.0:1769077230.628986:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077230.628988:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077230.628991:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077230.628999:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923836672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077230.629004:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077230.629007:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077230.629009:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077230.629012:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3919 00000100:00080000:3.0:1769077230.629015:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077230.629017:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077230.629020:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077230.629028:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077230.629030:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923836800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077230.629065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077230.629088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077230.629106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077230.948745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077231.652833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077231.972716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077232.676792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077232.996709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077233.700788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077233.700885:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769077233.700928:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077233.700949:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077233.700992:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077233.701004:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077233.701019:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077233.701031:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077233.701048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077230/real 1769077233] req@ffff969e0a7fd880 x1855007923836416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077233.701081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923836416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077233.701103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077233.701110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077233.701115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077233.701124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077230/real 1769077233] req@ffff969e0a7fca80 x1855007923836544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077233.701134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923836544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077233.701142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077233.701146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077233.701150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077233.701158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077230/real 1769077233] req@ffff969e0a7fe300 x1855007923836672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077233.701172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923836672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077233.701181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077233.701185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077233.701209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077233.701217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077230/real 1769077233] req@ffff969e0a7fd180 x1855007923836288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077233.701233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923836288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077233.701242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077233.701247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077233.701250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923836288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077233.701281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077230/real 1769077233] req@ffff969e0a7fc380 x1855007923836800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077233.701294:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923836800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077233.701303:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077233.701306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077233.701310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923836800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077234.020695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077234.724759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077235.044716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077235.748719:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077235.748733:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077235.748738:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077235.748751:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3924 00010000:00010000:3.0:1769077235.748761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077235.748782:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077235.748791:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077235.748846:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923836928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077235.748877:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077235.748897:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077235.748900:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077235.748904:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3924 00000100:00080000:2.0:1769077235.748909:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077235.748914:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077235.748925:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077235.748937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923836928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077235.748941:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923837056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077235.748955:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077235.748970:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077235.748972:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077235.748981:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3924 00000100:00080000:2.0:1769077235.748985:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077235.748989:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077235.748995:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077235.749009:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923837184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077235.749019:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077235.749029:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077235.749032:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077235.749036:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3924 00000800:00000100:1.0:1769077235.749043:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077235.749044:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077235.749052:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077235.749058:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077235.749072:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923837312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077235.749083:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077235.749089:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077235.749090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077235/real 1769077235] req@ffff969e0a46e300 x1855007923836928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077235.749092:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077235.749102:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3924 00000100:00080000:2.0:1769077235.749109:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077235.749120:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077235.749125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923836928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077235.749127:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077235.749141:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923837440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077235.749156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077235.749165:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077235.749172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923836928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077235.749221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077235.749254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077235.749269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077235/real 1769077235] req@ffff969e0a46ed80 x1855007923837056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077235.749283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923837056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077235.749291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077235.749295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077235.749298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077235.749305:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077235.749327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077235.749339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077235/real 1769077235] req@ffff969e0a46ce00 x1855007923837184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077235.749353:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923837184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077235.749362:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077235.749366:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077235.749369:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077235.749383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077235.749414:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077235.749425:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077235/real 1769077235] req@ffff969e0a46df80 x1855007923837312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077235.749438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923837312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077235.749448:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077235.749451:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077235.749454:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077235.749461:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077235.749482:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077235.749494:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077235/real 1769077235] req@ffff969e0a46ca80 x1855007923837440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077235.749508:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923837440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077235.749518:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077235.749521:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077235.749525:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077236.068680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077236.772856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077237.092694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077237.796824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077238.116649:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077238.820755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077239.140690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077239.844761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077240.164712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077240.868720:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077240.868730:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077240.868735:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077240.868745:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3930 00000100:00080000:2.0:1769077240.868756:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077240.868764:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077240.868775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077240.868800:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923837568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077240.868836:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077240.868839:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077240.868840:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077240.868843:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3930 00000100:00080000:2.0:1769077240.868846:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077240.868849:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077240.868859:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077240.868871:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923837696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077240.868883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923837568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077240.868895:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077240.868900:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077240.868903:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077240.868910:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3930 00000100:00080000:2.0:1769077240.868916:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077240.868920:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077240.868927:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077240.868944:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923837824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077240.868952:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077240.868955:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077240.868956:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077240.868961:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3930 00000800:00000100:1.0:1769077240.868965:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077240.868974:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077240.868979:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077240.868987:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077240.868998:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923837952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077240.869006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077240/real 1769077240] req@ffff969e0a46ea00 x1855007923837568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077240.869011:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077240.869030:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077240.869036:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077240.869045:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3930 00000100:00080000:2.0:1769077240.869048:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077240.869052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923837568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077240.869053:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077240.869064:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077240.869074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077240.869080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077240.869084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923837568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077240.869089:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923838080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077240.869127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077240.869156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077240.869171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077240/real 1769077240] req@ffff969e0a46f100 x1855007923837696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077240.869185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923837696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077240.869194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077240.869197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077240.869199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077240.869204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077240.869220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077240.869227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077240/real 1769077240] req@ffff969e0a46d500 x1855007923837824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077240.869248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923837824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077240.869254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077240.869257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077240.869259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077240.869264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077240.869279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077240.869285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077240/real 1769077240] req@ffff969e0a46f480 x1855007923837952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077240.869293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923837952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077240.869299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077240.869301:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077240.869303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923837952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077240.869307:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077240.869317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077240.869322:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077240/real 1769077240] req@ffff969e0a46d880 x1855007923838080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077240.869329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923838080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077240.869344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077240.869347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077240.869349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077241.188699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077241.892734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077242.212700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077242.916842:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077243.236712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077243.940759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077244.260736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077244.964747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077245.284668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077245.988757:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077245.988767:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077245.988771:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077245.988780:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3935 00000100:00080000:0.0:1769077245.988791:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769077245.988795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077245.988797:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077245.988833:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923838208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077245.988856:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077245.988873:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077245.988875:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077245.988879:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3935 00000100:00080000:0.0:1769077245.988883:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077245.988888:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077245.988889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923838208:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077245.988897:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077245.988910:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923838336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077245.988919:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077245.988921:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077245.988923:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077245.988926:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3935 00000100:00080000:0.0:1769077245.988937:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077245.988939:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077245.988943:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077245.988951:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923838464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077245.988959:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077245.988959:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077245.988964:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077245.988966:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077245.988969:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3935 00000100:00080000:0.0:1769077245.988973:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077245.988976:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077245.988985:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077245.988998:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077245/real 1769077245] req@ffff969e06a60000 x1855007923838208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077245.988999:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923838592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077245.989026:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077245.989029:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077245.989031:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077245.989034:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3935 00000100:00100000:1.0:1769077245.989037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923838208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077245.989039:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077245.989046:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077245.989052:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077245.989055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077245.989062:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923838720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077245.989062:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077245.989070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923838208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077245.989096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077245.989112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077245.989121:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077245/real 1769077245] req@ffff969e06a60a80 x1855007923838336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077245.989129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923838336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077245.989142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077245.989144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077245.989147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077245.989151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077245.989172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077245.989179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077245/real 1769077245] req@ffff969e06a63800 x1855007923838464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077245.989187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923838464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077245.989193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077245.989196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077245.989198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077245.989201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077245.989211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077245.989217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077245/real 1769077245] req@ffff969e06a60700 x1855007923838592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077245.989230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923838592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077245.989237:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077245.989239:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077245.989245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077245.989250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077245.989264:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077245.989270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077245/real 1769077245] req@ffff969e06a62d80 x1855007923838720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077245.989278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923838720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077245.989283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077245.989285:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077245.989287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077246.308714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077247.012757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077247.332719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077248.036778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077248.356708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077249.060788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077249.380696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077250.084802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077250.404673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077251.108697:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077251.108709:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077251.108715:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077251.108726:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3940 00010000:00010000:0.0:1769077251.108738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077251.108739:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077251.108749:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077251.108789:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923838848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077251.108815:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077251.108819:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077251.108820:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077251.108824:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3940 00000100:00080000:3.0:1769077251.108828:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077251.108832:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077251.108840:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077251.108848:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923838848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077251.108852:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923838976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077251.108883:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077251.108885:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077251.108887:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077251.108890:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3940 00000100:00080000:3.0:1769077251.108894:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077251.108896:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077251.108901:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077251.108911:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923839104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077251.108915:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077251.108924:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077251.108927:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077251.108928:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077251.108932:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3940 00000100:00080000:3.0:1769077251.108937:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077251.108939:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077251.108943:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077251.108951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077251/real 1769077251] req@ffff969e0a7fdf80 x1855007923838848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077251.108954:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923839232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077251.108978:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077251.108980:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077251.108982:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077251.108986:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3940 00000100:00100000:1.0:1769077251.108987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923838848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077251.108999:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077251.108999:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077251.109006:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077251.109010:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077251.109015:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077251.109019:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923839360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077251.109031:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923838848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077251.109054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077251.109074:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077251.109082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077251/real 1769077251] req@ffff969e0a7fea00 x1855007923838976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077251.109091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923838976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077251.109097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077251.109100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077251.109102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923838976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077251.109106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077251.109117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077251.109123:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077251/real 1769077251] req@ffff969e0a7fc700 x1855007923839104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077251.109130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923839104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077251.109135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077251.109137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077251.109139:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077251.109142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077251.109152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077251.109165:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077251/real 1769077251] req@ffff969e0a7fce00 x1855007923839232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077251.109173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923839232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077251.109186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077251.109188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077251.109190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077251.109194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077251.109207:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077251.109213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077251/real 1769077251] req@ffff969e0a7fe680 x1855007923839360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077251.109221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923839360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077251.109227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077251.109228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077251.109230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077251.428665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077252.132766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077252.452680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077253.156771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077253.476706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077254.180772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077254.500731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077255.204820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077255.524722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077256.228721:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077256.228733:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077256.228738:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077256.228748:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3945 00000100:00080000:0.0:1769077256.228759:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077256.228767:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077256.228779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077256.228812:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923839488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077256.228858:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077256.228861:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077256.228864:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077256.228869:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3945 00000100:00080000:0.0:1769077256.228883:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077256.228887:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077256.228899:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077256.228914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923839488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077256.228921:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923839616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077256.228934:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077256.228949:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077256.228951:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077256.228955:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3945 00000100:00080000:0.0:1769077256.228970:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077256.228979:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077256.228984:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077256.228995:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923839744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077256.229005:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077256.229009:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077256.229011:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077256.229015:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3945 00000800:00000100:1.0:1769077256.229018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077256.229030:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077256.229038:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077256.229065:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077256.229078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077256/real 1769077256] req@ffff969e06a61180 x1855007923839488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077256.229083:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923839872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077256.229097:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077256.229101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923839488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077256.229103:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077256.229108:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077256.229118:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3945 00000100:00080000:1.0:1769077256.229119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077256.229125:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077256.229126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077256.229133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923839488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077256.229136:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077256.229143:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077256.229156:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923840000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077256.229160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077256.229185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077256.229198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077256/real 1769077256] req@ffff969e06a62680 x1855007923839616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077256.229208:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923839616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077256.229214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077256.229216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077256.229218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077256.229222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077256.229349:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077256.229359:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077256/real 1769077256] req@ffff969e06a61f80 x1855007923839744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077256.229368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923839744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077256.229383:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077256.229386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077256.229388:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077256.229393:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077256.229408:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077256.229415:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077256/real 1769077256] req@ffff969e06a60380 x1855007923839872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077256.229422:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923839872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077256.229429:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077256.229430:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077256.229432:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923839872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077256.229443:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077256.229454:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077256.229460:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077256/real 1769077256] req@ffff969e06a61500 x1855007923840000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077256.229468:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923840000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077256.229473:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077256.229475:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077256.229484:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077256.548696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077257.252845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077257.572687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077258.277248:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077258.596690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077259.301059:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077259.620686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077260.324931:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077260.644674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077261.349029:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077261.349173:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077261.349183:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077261.349189:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077261.349203:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3950 00000100:00080000:2.0:1769077261.349217:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077261.349226:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077261.349272:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923840128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077261.349300:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077261.349315:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077261.349319:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077261.349325:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3950 00000100:00080000:2.0:1769077261.349331:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077261.349337:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077261.349349:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077261.349358:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923840128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077261.349369:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923840256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077261.349384:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077261.349388:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077261.349390:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077261.349394:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3950 00000100:00080000:2.0:1769077261.349399:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077261.349404:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077261.349410:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077261.349427:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923840384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077261.349438:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077261.349443:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077261.349444:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077261.349451:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3950 00000800:00000100:1.0:1769077261.349453:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077261.349460:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077261.349467:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077261.349480:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077261.349496:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923840512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077261.349506:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077261.349509:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077261.349511:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077261.349515:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3950 00000100:00080000:2.0:1769077261.349520:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077261.349524:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077261.349529:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077261.349540:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923840640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077261.349585:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077261/real 1769077261] req@ffff969e0a46c380 x1855007923840128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077261.349635:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923840128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077261.349648:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077261.349653:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077261.349658:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923840128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077261.349686:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077261.349714:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077261.349728:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077261/real 1769077261] req@ffff969e0a46f800 x1855007923840256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077261.349743:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923840256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077261.349752:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077261.349756:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077261.349769:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077261.349776:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077261.349800:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077261.349812:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077261/real 1769077261] req@ffff969e0a46d880 x1855007923840384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077261.349824:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923840384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077261.349833:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077261.349837:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077261.349842:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077261.349850:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077261.349870:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077261.349881:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077261/real 1769077261] req@ffff969e0a46f480 x1855007923840512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077261.349894:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923840512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077261.349903:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077261.349907:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077261.349911:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077261.349917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077261.349938:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077261.349957:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077261/real 1769077261] req@ffff969e0a46d500 x1855007923840640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077261.349970:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923840640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077261.349979:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077261.349983:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077261.349987:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077261.668674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077262.373009:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077262.692689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077263.397336:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077263.716713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077264.420777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077264.740711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077265.444827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077265.764786:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077266.468744:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077266.468754:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077266.468758:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077266.468767:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3955 00000100:00080000:3.0:1769077266.468778:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077266.468784:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077266.468804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077266.468821:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923840768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077266.468846:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077266.468849:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077266.468851:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077266.468854:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3955 00000100:00080000:3.0:1769077266.468856:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077266.468859:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077266.468883:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077266.468894:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923840896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077266.468904:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077266.468907:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077266.468910:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077266.468912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923840768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077266.468914:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3955 00000100:00080000:3.0:1769077266.468921:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077266.468927:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077266.468932:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077266.468947:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923841024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077266.468958:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077266.468962:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077266.468965:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077266.468969:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3955 00000100:00080000:3.0:1769077266.468984:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077266.468989:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077266.468995:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077266.468997:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077266.469010:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923841152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077266.469017:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077266.469020:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077266.469027:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077266.469031:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3955 00000100:00080000:3.0:1769077266.469035:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077266.469037:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077266/real 1769077266] req@ffff969e0a7fed80 x1855007923840768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077266.469041:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077266.469048:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077266.469075:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923841280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077266.469092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923840768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077266.469124:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077266.469132:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077266.469139:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923840768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077266.469175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077266.469223:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077266.469247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077266/real 1769077266] req@ffff969e0a7fe680 x1855007923840896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077266.469260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923840896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077266.469271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077266.469274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077266.469278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923840896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077266.469285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077266.469305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077266.469316:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077266/real 1769077266] req@ffff969e0a7fce00 x1855007923841024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077266.469330:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923841024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077266.469339:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077266.469343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077266.469347:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077266.469353:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077266.469369:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077266.469378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077266/real 1769077266] req@ffff969e0a7fc700 x1855007923841152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077266.469390:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923841152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077266.469406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077266.469410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077266.469414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077266.469420:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077266.469438:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077266.469448:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077266/real 1769077266] req@ffff969e0a7fea00 x1855007923841280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077266.469461:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923841280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077266.469471:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077266.469474:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077266.469477:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077266.788736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077267.492770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077267.812746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077268.516894:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077268.836834:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077269.540921:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077269.860778:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077270.564954:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077270.884818:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077271.588731:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077271.588746:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077271.588751:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077271.588764:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3960 00000100:00080000:3.0:1769077271.588779:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077271.588791:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077271.588848:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077271.588855:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923841408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077271.588904:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077271.588910:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077271.588913:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077271.588920:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3960 00000100:00080000:3.0:1769077271.588926:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077271.588931:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077271.588944:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077271.588951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923841408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077271.588968:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923841536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077271.589048:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077271.589054:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077271.589058:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077271.589066:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3960 00000800:00000100:1.0:1769077271.589071:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077271.589075:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077271.589083:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077271.589091:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077271.589108:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923841664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077271.589117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077271/real 1769077271] req@ffff969e0a7fdf80 x1855007923841408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077271.589127:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077271.589140:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077271.589143:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077271.589150:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3960 00000100:00080000:3.0:1769077271.589155:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077271.589160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923841408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077271.589163:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077271.589168:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077271.589187:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923841792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077271.589189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077271.589202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077271.589206:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077271.589210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923841408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077271.589212:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077271.589227:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077271.589233:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3960 00000100:00080000:3.0:1769077271.589247:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077271.589252:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077271.589258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841536:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769077271.589262:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077271.589285:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923841920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077271.589305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077271.589324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077271/real 1769077271] req@ffff969e0a7fc380 x1855007923841536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077271.589338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923841536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077271.589369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077271.589374:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077271.589378:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077271.589386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077271.589412:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077271.589437:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077271/real 1769077271] req@ffff969e0a7fd180 x1855007923841664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077271.589456:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923841664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077271.589467:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077271.589471:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077271.589474:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077271.589499:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077271.589523:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077271.589533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077271/real 1769077271] req@ffff969e0a7fe300 x1855007923841792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077271.589544:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923841792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077271.589564:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077271.589567:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077271.589570:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077271.589585:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077271.589628:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077271.589638:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077271/real 1769077271] req@ffff969e0a7fca80 x1855007923841920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077271.589649:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923841920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077271.589657:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077271.589660:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077271.589663:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923841920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077271.908759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077272.612853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077272.932730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077273.637017:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077273.956784:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077274.660971:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077274.980735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077275.684902:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077276.004814:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077276.708791:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077276.708802:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077276.708806:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077276.708826:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3965 00000100:00080000:2.0:1769077276.708837:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077276.708843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077276.708844:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077276.708883:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923842048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077276.708916:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077276.708919:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077276.708921:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077276.708924:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3965 00000100:00080000:2.0:1769077276.708940:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077276.708944:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077276.708953:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077276.708959:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923842048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077276.708968:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923842176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077276.708978:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077276.708980:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077276.708982:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077276.708984:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3965 00000100:00080000:2.0:1769077276.708987:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077276.708989:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077276.708993:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077276.709002:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923842304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077276.709009:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077276.709011:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077276.709012:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077276.709016:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3965 00000100:00080000:2.0:1769077276.709018:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077276.709020:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077276.709022:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077276.709027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077276.709031:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923842432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077276.709055:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077276.709058:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077276.709060:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077276.709062:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077276/real 1769077276] req@ffff969e0a46f100 x1855007923842048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077276.709065:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3965 00000100:00080000:2.0:1769077276.709072:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077276.709077:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077276.709081:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077276.709087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923842048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077276.709092:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923842560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077276.709102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077276.709109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077276.709114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923842048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077276.709134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077276.709149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077276.709157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077276/real 1769077276] req@ffff969e0a46ea00 x1855007923842176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077276.709164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923842176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077276.709170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077276.709172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077276.709175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077276.709178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077276.709188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077276.709194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077276/real 1769077276] req@ffff969e0a46ca80 x1855007923842304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077276.709209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923842304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077276.709215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077276.709217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077276.709219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077276.709223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077276.709236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077276.709242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077276/real 1769077276] req@ffff969e0a46df80 x1855007923842432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077276.709249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923842432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077276.709255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077276.709256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077276.709258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077276.709261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077276.709271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077276.709276:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077276/real 1769077276] req@ffff969e0a46ce00 x1855007923842560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077276.709283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923842560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077276.709312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077276.709314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077276.709316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077277.028686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077277.732833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077278.052762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077278.756827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077279.076793:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077279.780917:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077280.100719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077280.804881:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077281.124767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077281.828757:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077281.828769:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077281.828774:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077281.828786:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3971 00000100:00080000:2.0:1769077281.828836:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077281.828844:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077281.828917:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077281.828918:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923842688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077281.828956:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077281.828960:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077281.828962:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077281.828967:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3971 00000100:00080000:2.0:1769077281.828972:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077281.828978:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077281.829036:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077281.829039:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923842688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077281.829070:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923842816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077281.829096:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077281.829100:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077281.829103:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077281.829108:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3971 00000100:00080000:2.0:1769077281.829113:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077281.829118:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077281.829126:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077281.829141:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923842944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077281.829152:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077281.829156:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077281.829158:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077281.829161:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3971 00000100:00080000:2.0:1769077281.829165:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077281.829177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077281.829178:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077281.829190:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077281.829223:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923843072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077281.829234:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077281.829238:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077281.829250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077281/real 1769077281] req@ffff969e0a46ed80 x1855007923842688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077281.829250:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077281.829271:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3971 00000100:00080000:2.0:1769077281.829297:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077281.829309:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077281.829317:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077281.829326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923842688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077281.829349:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923843200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077281.829414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077281.829424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077281.829434:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923842688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077281.829493:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077281.829537:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077281.829551:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077281/real 1769077281] req@ffff969e0a46e300 x1855007923842816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077281.829562:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923842816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077281.829618:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077281.829622:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077281.829625:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077281.829632:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077281.829652:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077281.829669:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077281/real 1769077281] req@ffff969e0a46ce00 x1855007923842944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077281.829678:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923842944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077281.829685:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077281.829688:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077281.829690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923842944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077281.829695:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077281.829707:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077281.829714:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077281/real 1769077281] req@ffff969e0a46df80 x1855007923843072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077281.829731:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923843072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077281.829738:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077281.829740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077281.829749:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077281.829754:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077281.829771:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077281.829779:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077281/real 1769077281] req@ffff969e0a46ca80 x1855007923843200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077281.829791:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923843200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077281.829817:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077281.829822:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077281.829826:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077282.148887:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077282.852940:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077283.172746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077283.876863:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077284.196673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077284.900816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077285.220728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077285.924803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077286.244780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077286.948728:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077286.948739:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077286.948744:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077286.948756:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3976 00000100:00080000:2.0:1769077286.948769:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077286.948777:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077286.948812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077286.948817:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923843328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077286.948866:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077286.948870:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077286.948872:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077286.948878:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3976 00000100:00080000:2.0:1769077286.948883:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077286.948888:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077286.948898:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077286.948916:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923843456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077286.948917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923843328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077286.948991:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077286.948996:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077286.949000:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077286.949008:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3976 00000100:00080000:2.0:1769077286.949013:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077286.949018:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077286.949041:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077286.949057:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077286.949057:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923843584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077286.949072:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077286.949076:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077286.949079:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077286.949083:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3976 00000100:00080000:2.0:1769077286.949114:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077286.949118:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077286.949123:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077286.949130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077286/real 1769077286] req@ffff969e0a46ea00 x1855007923843328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077286.949138:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923843712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077286.949182:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077286.949186:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077286.949188:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077286.949192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923843328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077286.949197:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3976 00000100:00080000:2.0:1769077286.949210:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077286.949216:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077286.949219:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077286.949226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769077286.949229:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077286.949245:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923843840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077286.949247:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923843328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077286.949301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077286.949326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077286.949337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077286/real 1769077286] req@ffff969e0a46f100 x1855007923843456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077286.949347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923843456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077286.949354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077286.949357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077286.949359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077286.949364:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077286.949376:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077286.949382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077286/real 1769077286] req@ffff969e0a46d500 x1855007923843584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077286.949389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923843584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077286.949395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077286.949397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077286.949399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077286.949409:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077286.949422:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077286.949436:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077286/real 1769077286] req@ffff969e0a46f480 x1855007923843712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077286.949444:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923843712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077286.949451:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077286.949453:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077286.949455:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077286.949459:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077286.949477:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077286.949484:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077286/real 1769077286] req@ffff969e0a46d880 x1855007923843840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077286.949492:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923843840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077286.949498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077286.949501:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077286.949503:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923843840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077287.268754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077287.972855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077288.292709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077288.996912:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077289.316814:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077290.020875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077290.340757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077291.044823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077291.364734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077292.068720:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077292.068732:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077292.068738:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077292.068750:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3981 00000100:00080000:0.0:1769077292.068764:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077292.068773:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077292.068783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077292.068825:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923843968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077292.068849:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077292.068852:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077292.068854:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077292.068857:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3981 00000100:00100000:1.0:1769077292.068885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923843968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077292.068888:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077292.068896:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077292.068906:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077292.068920:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923844096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077292.068929:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077292.068932:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077292.068934:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077292.068936:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3981 00000100:00080000:0.0:1769077292.068947:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077292.068950:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077292.068953:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077292.068960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077292.068965:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923844224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077292.068980:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077292.068984:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077292.068987:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077292.068991:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3981 00000100:00080000:0.0:1769077292.068996:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077292.069001:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077292.069008:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077292.069008:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077292/real 1769077292] req@ffff969e06a63100 x1855007923843968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077292.069030:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923844352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077292.069030:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923843968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077292.069065:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077292.069069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077292.069073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923843968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077292.069092:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077292.069096:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077292.069099:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077292.069101:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844096:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769077292.069107:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3981 00000100:00080000:0.0:1769077292.069113:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077292.069117:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077292.069122:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077292.069134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077292.069136:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923844480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077292.069156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077292/real 1769077292] req@ffff969e06a61880 x1855007923844096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077292.069179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923844096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077292.069189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077292.069193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077292.069197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077292.069204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077292.069227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077292.069239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077292/real 1769077292] req@ffff969e06a61500 x1855007923844224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077292.069252:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923844224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077292.069271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077292.069274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077292.069276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077292.069281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077292.069297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077292.069304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077292/real 1769077292] req@ffff969e06a60380 x1855007923844352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077292.069312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923844352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077292.069318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077292.069321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077292.069323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077292.069333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077292.069344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077292.069349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077292/real 1769077292] req@ffff969e06a61f80 x1855007923844480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077292.069356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923844480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077292.069362:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077292.069364:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077292.069371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077292.388738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077293.092833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077293.412777:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077294.116862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077294.436688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077295.140781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077295.460694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077296.164753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077296.484704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077297.188695:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077297.188710:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077297.188716:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077297.188729:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3986 00010000:00010000:3.0:1769077297.188742:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077297.188744:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077297.188755:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077297.188800:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923844608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077297.188824:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077297.188829:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077297.188831:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077297.188835:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3986 00000100:00080000:2.0:1769077297.188849:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077297.188853:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077297.188864:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077297.188866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923844608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077297.188878:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923844736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077297.188889:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077297.188912:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077297.188914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077297.188918:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3986 00000100:00080000:2.0:1769077297.188921:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077297.188925:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077297.188929:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077297.188942:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923844864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077297.188953:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077297.188957:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077297.188959:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077297.188962:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3986 00000100:00080000:2.0:1769077297.188966:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077297.188970:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077297.188976:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077297.188982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077297.188988:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923844992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077297.188998:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077297.189001:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077297.189003:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077297.189007:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3986 00000100:00080000:2.0:1769077297.189011:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077297.189015:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844864:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077297.189015:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077297.189027:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077297.189039:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923845120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077297.189042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077297.189068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077297.508703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077298.212734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077298.532688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077299.236780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077299.556679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077300.260797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769077300.260939:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769077300.260988:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769077300.261013:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077300.261056:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077300.261067:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077300.261074:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077300.261079:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077300.261352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077297/real 1769077300] req@ffff969e0a46d500 x1855007923845120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077300.261385:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923845120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077300.261400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077300.261407:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077300.261412:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077300.261421:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077297/real 1769077300] req@ffff969e0a46f480 x1855007923844992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077300.261459:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923844992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077300.261469:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077300.261473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077300.261477:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077300.261484:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077297/real 1769077300] req@ffff969e0a46c380 x1855007923844736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077300.261538:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923844736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077300.261547:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077300.261552:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077300.261556:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077300.261564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077297/real 1769077300] req@ffff969e0a46d880 x1855007923844864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077300.261577:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923844864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077300.261585:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077300.261615:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077300.261618:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923844864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077300.261625:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077297/real 1769077300] req@ffff969e0a46f800 x1855007923844608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077300.261648:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923844608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077300.261658:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077300.261663:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077300.261667:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923844608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077300.580703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077301.284779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077301.604735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077302.308802:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769077302.308802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077302.308812:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077302.308817:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077302.308827:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3991 00000100:00080000:0.0:1769077302.308838:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077302.308845:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077302.308888:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923845248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077302.308942:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077302.308946:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077302.308949:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077302.308956:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3991 00000100:00080000:0.0:1769077302.308963:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077302.308973:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077302.308985:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077302.309006:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923845376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077302.309009:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923845248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077302.309037:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077302.309042:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077302.309045:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077302.309052:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3991 00000100:00080000:0.0:1769077302.309059:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077302.309066:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077302.309072:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077302.309088:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923845504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077302.309100:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077302.309104:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077302.309107:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077302.309113:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3991 00000800:00000100:1.0:1769077302.309121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077302.309131:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077302.309142:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077302.309151:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077302.309176:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923845632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077302.309179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077302/real 1769077302] req@ffff969e06a62680 x1855007923845248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077302.309207:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077302.309216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923845248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077302.309217:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077302.309223:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077302.309238:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3991 00000100:00080000:0.0:1769077302.309250:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077302.309250:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077302.309260:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077302.309266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:0.0:1769077302.309274:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077302.309275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923845248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077302.309303:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923845760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077302.309324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077302.309360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077302.309376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077302/real 1769077302] req@ffff969e06a61180 x1855007923845376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077302.309391:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923845376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077302.309418:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077302.309422:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077302.309426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077302.309435:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077302.309479:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077302.309493:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077302/real 1769077302] req@ffff969e06a62d80 x1855007923845504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077302.309506:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923845504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077302.309515:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077302.309519:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077302.309523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077302.309531:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077302.309553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077302.309564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077302/real 1769077302] req@ffff969e06a60700 x1855007923845632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077302.309577:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923845632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077302.309587:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077302.309626:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077302.309638:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077302.309662:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077302.309685:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077302.309697:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077302/real 1769077302] req@ffff969e06a63800 x1855007923845760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077302.309710:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923845760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077302.309720:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077302.309723:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077302.309727:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923845760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077302.628692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077303.332800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077303.652668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077304.356758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077304.676695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077305.380770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077305.700715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077306.404759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077306.724766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077307.428683:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077307.428695:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077307.428699:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077307.428710:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 3996 00000100:00080000:0.0:1769077307.428722:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077307.428729:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077307.428772:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923845888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769077307.428778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077307.428798:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077307.428825:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077307.428826:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077307.428833:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3996 00000100:00100000:1.0:1769077307.428833:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923845888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077307.428839:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077307.428844:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077307.428853:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077307.428865:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923846016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077307.428874:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077307.428877:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077307.428878:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077307.428882:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3996 00000100:00080000:0.0:1769077307.428893:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077307.428896:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077307.428899:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077307.428911:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923846144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077307.428911:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077307.428922:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077307.428925:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077307.428927:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077307.428930:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3996 00000100:00080000:0.0:1769077307.428932:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077307.428935:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077307.428944:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077307.428952:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923846272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077307.428959:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077307.428961:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077307/real 1769077307] req@ffff969e06a60a80 x1855007923845888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077307.428963:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077307.428972:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077307.428976:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 3996 00000100:00080000:0.0:1769077307.428980:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077307.428983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923845888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077307.428984:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077307.428992:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077307.429001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077307.429006:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923846400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077307.429010:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077307.429022:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923845888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077307.429048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077307.429067:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077307.429084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077307/real 1769077307] req@ffff969e06a60000 x1855007923846016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077307.429094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923846016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077307.429101:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077307.429103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077307.429106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077307.429110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077307.429123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077307.429130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077307/real 1769077307] req@ffff969e06a63800 x1855007923846144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077307.429138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923846144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077307.429144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077307.429146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077307.429148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077307.429152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077307.429163:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077307.429187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077307/real 1769077307] req@ffff969e06a60700 x1855007923846272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077307.429201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923846272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077307.429211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077307.429215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077307.429219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077307.429226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077307.429248:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077307.429256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077307/real 1769077307] req@ffff969e06a62d80 x1855007923846400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077307.429265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923846400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077307.429271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077307.429274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077307.429276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077307.748725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077308.452767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077308.772708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077309.476767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077309.796724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077310.500804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077310.820726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077311.524758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077311.844720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077312.548723:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077312.548734:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077312.548738:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077312.548751:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4001 00000100:00080000:0.0:1769077312.548764:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077312.548771:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077312.548804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077312.548815:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923846528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077312.548856:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077312.548861:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077312.548863:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077312.548869:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4001 00000100:00080000:0.0:1769077312.548892:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077312.548905:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077312.548905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923846528:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077312.548922:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077312.548937:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923846656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077312.548948:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077312.548950:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077312.548952:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077312.548955:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4001 00000100:00080000:0.0:1769077312.548959:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077312.548961:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077312.548966:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077312.548976:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923846784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077312.548987:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077312.548991:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077312.548993:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077312.549001:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077312.549004:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4001 00000100:00080000:0.0:1769077312.549010:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077312.549018:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077312.549026:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077312.549042:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923846912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077312.549062:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077312.549062:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077312/real 1769077312] req@ffff969e06a61180 x1855007923846528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077312.549079:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077312.549085:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077312.549101:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4001 00000100:00080000:0.0:1769077312.549108:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077312.549112:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077312.549117:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077312.549124:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923846528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077312.549128:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923847040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077312.549197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077312.549205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077312.549213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923846528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077312.549244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077312.549272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077312.549325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077312/real 1769077312] req@ffff969e06a62680 x1855007923846656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077312.549340:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923846656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077312.549350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077312.549354:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077312.549356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077312.549362:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077312.549379:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077312.549387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077312/real 1769077312] req@ffff969e06a61f80 x1855007923846784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077312.549396:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923846784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077312.549415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077312.549419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077312.549422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077312.549429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077312.549453:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077312.549465:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077312/real 1769077312] req@ffff969e06a60380 x1855007923846912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077312.549478:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923846912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077312.549485:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077312.549488:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077312.549490:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923846912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077312.549495:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077312.549510:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077312.549518:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077312/real 1769077312] req@ffff969e06a61500 x1855007923847040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077312.549526:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923847040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077312.549531:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077312.549533:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077312.549541:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077312.868690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077313.572782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077313.892713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077314.596791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077314.916752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077315.620807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077315.940724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077316.644737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077316.964720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077317.668746:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077317.668759:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077317.668765:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077317.668777:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4006 00010000:00010000:2.0:1769077317.668780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077317.668797:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077317.668804:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077317.668847:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923847168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077317.668871:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077317.668875:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077317.668877:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077317.668881:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4006 00000100:00080000:0.0:1769077317.668884:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077317.668889:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077317.668907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923847168:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077317.668915:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077317.668932:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923847296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077317.668944:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077317.668948:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077317.668950:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077317.668953:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4006 00000100:00080000:0.0:1769077317.668971:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077317.668975:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077317.668979:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077317.668987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077317.668992:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923847424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077317.669002:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077317.669006:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077317.669008:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077317.669012:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4006 00000100:00080000:0.0:1769077317.669016:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077317.669019:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077317.669024:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077317.669026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077317/real 1769077317] req@ffff969e06a61880 x1855007923847168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077317.669043:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923847552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077317.669049:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923847168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077317.669058:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077317.669066:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077317.669069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077317.669074:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077317.669078:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769077317.669083:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4006 00000100:00100000:1.0:1769077317.669087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923847168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077317.669089:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077317.669097:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077317.669103:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077317.669114:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923847680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077317.669117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077317.669146:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077317.669158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077317/real 1769077317] req@ffff969e06a63100 x1855007923847296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077317.669169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923847296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077317.669178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077317.669181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077317.669191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077317.669197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077317.669246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077317.669256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077317/real 1769077317] req@ffff969e06a61500 x1855007923847424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077317.669267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923847424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077317.669285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077317.669292:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077317.669296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077317.669302:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077317.669319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077317.669327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077317/real 1769077317] req@ffff969e06a60380 x1855007923847552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077317.669338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923847552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077317.669347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077317.669350:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077317.669353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077317.669367:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077317.669380:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077317.669397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077317/real 1769077317] req@ffff969e06a61f80 x1855007923847680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077317.669408:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923847680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077317.669417:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077317.669419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077317.669422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077317.988705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077318.692726:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077319.012673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077319.716774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077320.036669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077320.740777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077321.060669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077321.764782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077322.084669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077322.788717:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077322.788732:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077322.788738:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077322.788751:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4011 00000100:00080000:3.0:1769077322.788765:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077322.788774:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077322.788790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077322.788833:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923847808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077322.788862:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077322.788865:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077322.788867:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077322.788872:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4011 00000100:00080000:3.0:1769077322.788876:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077322.788880:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077322.788892:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077322.788906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923847936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077322.788917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923847808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077322.788919:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077322.788927:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077322.788929:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077322.788933:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4011 00000100:00080000:3.0:1769077322.788937:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077322.788941:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077322.788946:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077322.788978:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923848064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077322.788987:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077322.788991:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077322.788993:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077322.788999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077322.788999:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4011 00000100:00080000:3.0:1769077322.789006:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077322.789011:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077322.789016:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077322.789028:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923848192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077322.789038:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077322.789041:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077322.789043:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077322.789049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077322/real 1769077322] req@ffff969e0a7fd880 x1855007923847808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077322.789049:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4011 00000100:00080000:3.0:1769077322.789054:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077322.789061:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077322.789070:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077322.789082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923847808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077322.789084:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923848320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077322.789098:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077322.789107:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077322.789145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923847808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077322.789171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077322.789193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077322.789213:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077322/real 1769077322] req@ffff969e0a7ffb80 x1855007923847936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077322.789226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923847936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077322.789235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077322.789238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077322.789242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923847936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077322.789248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077322.789268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077322.789279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077322/real 1769077322] req@ffff969e0a7fdc00 x1855007923848064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077322.789292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923848064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077322.789300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077322.789303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077322.789306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077322.789313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077322.789332:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077322.789343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077322/real 1769077322] req@ffff969e0a7ff800 x1855007923848192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077322.789355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923848192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077322.789370:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077322.789374:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077322.789377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077322.789392:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077322.789418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077322.789428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077322/real 1769077322] req@ffff969e0a7fc000 x1855007923848320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077322.789439:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923848320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077322.789447:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077322.789450:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077322.789454:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077323.108661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077323.812767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077324.132705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077324.837801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077325.156685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077325.860935:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077326.180661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077326.885841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077327.204690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077327.909278:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077327.909367:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077327.909376:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077327.909381:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077327.909392:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4017 00000100:00080000:2.0:1769077327.909412:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077327.909420:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077327.909458:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923848448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077327.909487:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077327.909490:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077327.909493:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077327.909498:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4017 00000100:00080000:2.0:1769077327.909502:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077327.909507:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077327.909522:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923848448:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077327.909523:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077327.909541:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923848576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077327.909570:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077327.909574:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077327.909575:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077327.909580:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4017 00000100:00080000:2.0:1769077327.909586:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077327.909629:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077327.909635:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077327.909647:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077327.909653:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923848704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077327.909664:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077327.909668:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077327.909670:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077327.909674:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4017 00000100:00080000:2.0:1769077327.909679:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077327.909683:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077327.909687:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077327.909689:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077327/real 1769077327] req@ffff969e0a46f100 x1855007923848448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077327.909698:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923848832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077327.909716:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077327.909722:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077327.909724:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077327.909728:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4017 00000100:00100000:0.0:1769077327.909734:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923848448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077327.909741:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077327.909750:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769077327.909753:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769077327.909755:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769077327.909761:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077327.909766:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923848448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077327.909771:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923848960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077327.909803:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077327.909829:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077327.909842:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077327/real 1769077327] req@ffff969e0a46ea00 x1855007923848576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077327.909866:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923848576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077327.909874:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077327.909876:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077327.909880:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077327.909886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077327.909905:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077327.909915:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077327/real 1769077327] req@ffff969e0a46ca80 x1855007923848704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077327.909925:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923848704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077327.909932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077327.909935:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077327.909938:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077327.909943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077327.909960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077327.909970:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077327/real 1769077327] req@ffff969e0a46df80 x1855007923848832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077327.909980:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923848832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077327.910000:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077327.910004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077327.910007:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077327.910013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077327.910050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077327.910061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077327/real 1769077327] req@ffff969e0a46ce00 x1855007923848960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077327.910077:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923848960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077327.910086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077327.910089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077327.910092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923848960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077328.228706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077328.933364:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077329.252723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077329.956731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077330.276707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077330.980797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077331.300725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077332.004720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077332.324670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077333.028746:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077333.028756:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077333.028760:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077333.028770:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4022 00000100:00080000:3.0:1769077333.028780:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077333.028786:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077333.028814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077333.028853:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923849088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077333.028886:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077333.028891:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077333.028894:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077333.028900:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4022 00000100:00080000:3.0:1769077333.028905:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077333.028910:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077333.028919:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923849088:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077333.028920:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077333.029008:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077333.029037:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077333/real 1769077333] req@ffff969e0a7fd500 x1855007923849088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077333.029057:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923849216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077333.029088:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077333.029091:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077333.029093:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077333.029097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923849088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077333.029100:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4022 00000100:00080000:0.0:1769077333.029119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077333.029124:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077333.029126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077333.029131:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077333.029132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923849088:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077333.029140:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077333.029152:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923849344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077333.029158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849216:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769077333.029165:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077333.029169:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077333.029170:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077333.029174:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4022 00000100:00080000:3.0:1769077333.029177:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077333.029180:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077333.029184:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077333.029194:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923849472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077333.029195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077333.029204:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077333.029208:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077333.029210:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077333.029213:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4022 00000100:00100000:0.0:1769077333.029217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077333/real 1769077333] req@ffff969e0a7fc000 x1855007923849216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077333.029218:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077333.029223:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077333.029243:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077333.029249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923849216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077333.029254:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923849600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077333.029275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077333.029284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077333.029292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077333.029308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077333.029325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077333.029333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077333/real 1769077333] req@ffff969e0a7ff800 x1855007923849344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077333.029341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923849344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077333.029355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077333.029358:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077333.029360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077333.029364:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077333.029385:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077333.029391:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077333/real 1769077333] req@ffff969e0a7fdc00 x1855007923849472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077333.029398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923849472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077333.029404:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077333.029406:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077333.029407:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077333.029411:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077333.029420:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077333.029425:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077333/real 1769077333] req@ffff969e0a7ffb80 x1855007923849600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077333.029433:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923849600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077333.029452:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077333.029455:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077333.029466:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077333.348689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077334.052760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077334.372670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077335.076772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077335.396705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077336.100798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077336.420697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077337.124846:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077337.444728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077338.148714:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077338.148727:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077338.148732:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077338.148754:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4027 00000100:00080000:2.0:1769077338.148767:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077338.148775:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077338.148788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077338.148816:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923849728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077338.148845:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077338.148850:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077338.148852:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077338.148870:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4027 00000100:00080000:2.0:1769077338.148874:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077338.148880:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077338.148890:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077338.148895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923849728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077338.148909:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923849856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077338.148922:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077338.148944:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077338.148946:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077338.148951:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4027 00000100:00080000:2.0:1769077338.148956:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077338.148961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077338.148965:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077338.148972:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077338.148988:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923849984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077338.148995:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077338/real 1769077338] req@ffff969e0a46e300 x1855007923849728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077338.149006:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077338.149011:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077338.149014:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077338.149017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923849728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077338.149022:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4027 00000100:00080000:2.0:1769077338.149029:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077338.149036:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769077338.149039:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769077338.149043:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769077338.149048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077338.149055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923849728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077338.149062:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923850112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077338.149074:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077338.149078:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077338.149080:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077338.149087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849856:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077338.149087:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4027 00000100:00080000:2.0:1769077338.149094:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077338.149101:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077338.149115:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769077338.149122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077338.149133:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923850240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077338.149140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077338/real 1769077338] req@ffff969e0a46ed80 x1855007923849856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077338.149160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923849856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077338.149173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077338.149176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077338.149179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077338.149184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077338.149199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077338.149207:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077338/real 1769077338] req@ffff969e0a46ce00 x1855007923849984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077338.149214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923849984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077338.149220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077338.149222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077338.149224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923849984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077338.149227:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077338.149246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077338.149253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077338/real 1769077338] req@ffff969e0a46df80 x1855007923850112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077338.149260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923850112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077338.149266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077338.149268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077338.149275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077338.149289:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077338.149303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077338.149310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077338/real 1769077338] req@ffff969e0a46ca80 x1855007923850240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077338.149317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923850240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077338.149323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077338.149325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077338.149327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077338.468710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077339.172764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077339.492722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077340.196809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077340.516717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077341.220781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077341.540710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077342.244834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077342.564693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077343.268645:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077343.268655:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077343.268659:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077343.268668:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4032 00000100:00080000:2.0:1769077343.268679:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077343.268685:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077343.268718:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923850368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077343.268751:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077343.268754:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077343.268756:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769077343.268758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077343.268760:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4032 00000100:00080000:2.0:1769077343.268765:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077343.268785:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077343.268793:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077343.268807:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923850496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077343.268815:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077343.268818:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077343.268820:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077343.268822:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4032 00000100:00100000:1.0:1769077343.268839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923850368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077343.268840:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077343.268848:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077343.268853:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077343.268863:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923850624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077343.268871:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077343.268874:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077343.268875:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077343.268878:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4032 00000100:00080000:2.0:1769077343.268880:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077343.268883:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077343.268886:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077343.268893:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923850752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077343.268899:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077343.268902:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077343.268903:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077343.268905:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4032 00000100:00080000:2.0:1769077343.268907:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077343.268910:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077343.268914:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769077343.268916:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077343.268924:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923850880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077343.268952:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077343/real 1769077343] req@ffff969e0a46ea00 x1855007923850368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077343.268967:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923850368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077343.268975:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077343.268986:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077343.268989:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923850368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077343.268995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077343.269012:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077343.269021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077343/real 1769077343] req@ffff969e0a46f100 x1855007923850496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077343.269029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923850496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077343.269035:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077343.269037:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077343.269039:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077343.269059:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077343.269070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077343.269076:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077343/real 1769077343] req@ffff969e0a46f800 x1855007923850624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077343.269084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923850624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077343.269089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077343.269090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077343.269092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077343.269095:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077343.269115:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077343.269121:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077343/real 1769077343] req@ffff969e0a46d880 x1855007923850752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077343.269128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923850752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077343.269134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077343.269135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077343.269137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077343.269141:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077343.269150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077343.269156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077343/real 1769077343] req@ffff969e0a46c380 x1855007923850880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077343.269162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923850880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077343.269168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077343.269169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077343.269171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923850880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077343.588703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077344.292737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077344.612696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077345.316745:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077345.636716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077346.340833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077346.660759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077347.364773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077347.684662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077348.388747:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077348.388758:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077348.388764:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077348.388773:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4037 00000100:00080000:2.0:1769077348.388786:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077348.388792:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077348.388831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077348.388846:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923851008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077348.388873:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077348.388876:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077348.388877:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077348.388881:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4037 00000100:00080000:2.0:1769077348.388883:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077348.388886:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077348.388895:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077348.388905:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923851136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077348.388913:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077348.388916:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077348.388917:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077348.388922:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4037 00000100:00080000:2.0:1769077348.388924:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077348.388926:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077348.388930:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923851008:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077348.388948:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077348.388962:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923851264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077348.388970:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077348.388973:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077348.388974:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077348.388976:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4037 00000100:00080000:2.0:1769077348.388979:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077348.388982:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077348.388985:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077348.388993:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923851392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077348.389000:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077348.389003:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077348.389005:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077348.389009:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4037 00000800:00000100:1.0:1769077348.389010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077348.389012:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077348.389016:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077348.389019:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077348.389027:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923851520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077348.389052:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077348/real 1769077348] req@ffff969e0a46f480 x1855007923851008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077348.389081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923851008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077348.389089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077348.389093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077348.389096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923851008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077348.389124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077348.389143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077348.389152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077348/real 1769077348] req@ffff969e0a46d500 x1855007923851136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077348.389160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923851136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077348.389166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077348.389168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077348.389171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077348.389174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077348.389184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077348.389190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077348/real 1769077348] req@ffff969e0a46c380 x1855007923851264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077348.389197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923851264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077348.389212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077348.389214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077348.389216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077348.389221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077348.389235:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077348.389241:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077348/real 1769077348] req@ffff969e0a46d880 x1855007923851392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077348.389249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923851392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077348.389254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077348.389256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077348.389258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077348.389261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077348.389271:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077348.389285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077348/real 1769077348] req@ffff969e0a46f800 x1855007923851520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077348.389293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923851520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077348.389299:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077348.389301:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077348.389307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077348.708738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077349.412836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077349.732686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077350.436759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077350.756744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077351.460741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077351.780773:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077352.484759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077352.804721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077353.508721:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077353.508731:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077353.508736:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077353.508746:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4042 00000100:00080000:3.0:1769077353.508756:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077353.508763:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077353.508792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077353.508803:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923851648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077353.508829:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077353.508831:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077353.508832:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077353.508836:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4042 00000100:00080000:3.0:1769077353.508838:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077353.508841:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077353.508850:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077353.508858:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923851776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077353.508866:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077353.508870:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769077353.508871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923851648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077353.508872:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077353.508880:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4042 00000100:00080000:3.0:1769077353.508882:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077353.508885:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077353.508905:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077353.508915:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923851904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077353.508922:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077353.508925:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077353.508926:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077353.508928:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4042 00000100:00080000:3.0:1769077353.508931:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077353.508934:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077353.508937:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077353.508944:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923852032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077353.508951:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077353.508957:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077353.508960:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077353.508962:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077353.508966:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4042 00000100:00080000:3.0:1769077353.508976:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077353.508979:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077353.508983:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077353.508988:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077353/real 1769077353] req@ffff969e0a7fd880 x1855007923851648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077353.508994:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923852160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077353.509019:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923851648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077353.509030:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077353.509034:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077353.509037:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923851648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077353.509058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077353.509073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077353.509081:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077353/real 1769077353] req@ffff969e0a7fca80 x1855007923851776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077353.509089:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923851776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077353.509094:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077353.509096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077353.509099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077353.509102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077353.509112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077353.509127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077353/real 1769077353] req@ffff969e0a7fe300 x1855007923851904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077353.509135:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923851904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077353.509141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077353.509143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077353.509145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923851904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077353.509149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077353.509160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077353.509166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077353/real 1769077353] req@ffff969e0a7fd180 x1855007923852032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077353.509173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923852032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077353.509179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077353.509180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077353.509182:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077353.509185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077353.509195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077353.509200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077353/real 1769077353] req@ffff969e0a7fc380 x1855007923852160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077353.509207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923852160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077353.509217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077353.509220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077353.509222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077353.828759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077354.532817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077354.852805:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077355.556796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077355.876735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077356.580797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077356.900686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077357.604810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077357.924713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077358.628726:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077358.628739:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077358.628744:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077358.628753:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4047 00000100:00080000:0.0:1769077358.628765:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077358.628773:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077358.628784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077358.628817:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923852288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077358.628859:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077358.628862:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077358.628863:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077358.628867:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4047 00000100:00080000:0.0:1769077358.628874:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077358.628878:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077358.628888:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077358.628899:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923852416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077358.628905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923852288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077358.628909:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077358.628916:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077358.628918:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077358.628921:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4047 00000100:00080000:0.0:1769077358.628923:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077358.628925:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077358.628928:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077358.628937:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923852544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077358.628943:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077358.628945:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077358.628946:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077358.628949:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4047 00000100:00080000:0.0:1769077358.628957:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077358.628960:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077358.628965:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077358.628972:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923852672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077358.628979:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077358.628982:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077358.628983:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077358.628985:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4047 00000100:00080000:0.0:1769077358.628987:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077358.628990:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077358.628993:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077358.628995:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077358.629003:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923852800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077358.629041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077358/real 1769077358] req@ffff969e06a62680 x1855007923852288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077358.629064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923852288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077358.629076:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077358.629081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077358.629086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923852288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077358.629111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077358.629134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077358.629147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077358/real 1769077358] req@ffff969e06a61180 x1855007923852416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077358.629160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923852416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077358.629170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077358.629174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077358.629178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077358.629191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077358.629211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077358.629221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077358/real 1769077358] req@ffff969e06a62d80 x1855007923852544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077358.629234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923852544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077358.629242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077358.629246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077358.629249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077358.629256:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077358.629277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077358.629288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077358/real 1769077358] req@ffff969e06a60700 x1855007923852672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077358.629301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923852672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077358.629311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077358.629315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077358.629318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077358.629325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077358.629344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077358.629356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077358/real 1769077358] req@ffff969e06a63800 x1855007923852800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077358.629376:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923852800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077358.629387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077358.629390:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077358.629394:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923852800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077358.948735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077359.652809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077359.972719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077360.676773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077360.996675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077361.700772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077362.020668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077362.724736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077363.044737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077363.748718:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077363.748730:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077363.748749:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769077363.748751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077363.748758:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4052 00000100:00080000:2.0:1769077363.748769:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077363.748775:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077363.748804:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923852928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077363.748827:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077363.748829:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077363.748831:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077363.748834:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4052 00000100:00080000:2.0:1769077363.748837:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077363.748840:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077363.748848:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077363.748857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923852928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077363.748860:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923853056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077363.748871:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077363.748873:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077363.748875:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077363.748877:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4052 00000100:00080000:2.0:1769077363.748880:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077363.748882:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077363.748885:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077363.748894:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923853184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077363.748909:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077363.748912:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077363.748914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077363.748917:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4052 00000100:00080000:2.0:1769077363.748920:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077363.748923:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077363.748926:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077363.748936:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923853312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077363.748942:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077363.748945:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077363.748947:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077363.748950:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4052 00000100:00080000:2.0:1769077363.748953:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077363.748955:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077363.748958:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077363.748965:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923853440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077363.748968:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077363.749008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077363.749034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077363.749051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077364.068670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077364.772747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077365.092731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077365.796800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077366.116693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077366.820740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769077366.820882:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769077366.820924:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769077366.820944:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077366.821014:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077366.821025:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077366.821034:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077366.821043:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077366.821100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077363/real 1769077366] req@ffff969e0a46ce00 x1855007923853440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077366.821131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923853440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077366.821147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077366.821152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077366.821157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077366.821166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077363/real 1769077366] req@ffff969e0a46ea00 x1855007923853056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077366.821198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923853056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077366.821207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077366.821211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077366.821216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077366.821224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077363/real 1769077366] req@ffff969e0a46ca80 x1855007923853184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077366.821238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923853184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077366.821247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077366.821251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077366.821254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077366.821261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077363/real 1769077366] req@ffff969e0a46df80 x1855007923853312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077366.821272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923853312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077366.821281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077366.821284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077366.821287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077366.821295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077363/real 1769077366] req@ffff969e0a46f100 x1855007923852928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077366.821320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923852928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077366.821327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077366.821332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077366.821335:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923852928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077367.140682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077367.844786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077368.164691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077368.868686:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077368.868697:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077368.868701:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077368.868712:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4058 00010000:00010000:0.0:1769077368.868725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077368.868725:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077368.868735:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077368.868773:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923853568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077368.868800:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077368.868803:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077368.868804:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077368.868807:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4058 00000100:00080000:3.0:1769077368.868810:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077368.868813:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077368.868823:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077368.868836:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923853696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077368.868862:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077368.868875:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923853568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077368.868902:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077368.868905:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077368.868911:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4058 00000100:00080000:3.0:1769077368.868917:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077368.868922:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077368.868940:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077368.868957:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923853824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077368.868965:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077368.868975:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077368.868980:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077368.868982:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077368.868988:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4058 00000100:00080000:3.0:1769077368.868994:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077368.869000:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077368.869007:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077368.869014:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077368/real 1769077368] req@ffff969e0a7fdf80 x1855007923853568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077368.869031:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923853952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077368.869040:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923853568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077368.869045:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077368.869054:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077368.869060:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077368.869064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077368.869066:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4058 00000100:00080000:1.0:1769077368.869074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077368.869075:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077368.869083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923853568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077368.869085:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077368.869100:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077368.869119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077368.869122:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923854080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077368.869156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077368.869173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077368/real 1769077368] req@ffff969e0a7fea00 x1855007923853696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077368.869186:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923853696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077368.869195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077368.869208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077368.869212:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077368.869225:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077368.869249:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077368.869260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077368/real 1769077368] req@ffff969e0a7fc700 x1855007923853824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077368.869272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923853824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077368.869280:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077368.869284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077368.869287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077368.869294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077368.869314:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077368.869332:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077368/real 1769077368] req@ffff969e0a7fce00 x1855007923853952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077368.869382:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923853952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077368.869391:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077368.869395:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077368.869399:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923853952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077368.869417:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077368.869450:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077368.869462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077368/real 1769077368] req@ffff969e0a7fe680 x1855007923854080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077368.869475:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923854080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077368.869485:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077368.869490:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077368.869497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077369.188739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077369.892864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077370.212703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077370.916800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077371.236694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077371.940814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077372.260748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077372.964764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077373.284725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077373.988732:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077373.988741:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077373.988745:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077373.988757:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4063 00000100:00080000:3.0:1769077373.988770:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077373.988778:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077373.988814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077373.988822:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923854208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077373.988854:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077373.988857:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077373.988858:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077373.988863:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4063 00000100:00080000:3.0:1769077373.988869:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077373.988873:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077373.988882:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077373.988895:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923854208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077373.988898:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923854336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077373.988916:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077373.988919:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077373.988920:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077373.988924:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4063 00000100:00080000:3.0:1769077373.988941:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077373.988945:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077373.988950:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077373.988961:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923854464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077373.988968:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077373.988970:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077373.988972:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077373.988976:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4063 00000100:00080000:3.0:1769077373.988981:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077373.988984:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077373.988988:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077373.988992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077373.988999:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923854592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077373.989008:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077373.989011:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077373.989012:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077373.989017:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4063 00000100:00080000:3.0:1769077373.989020:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077373.989023:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077373.989029:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077373.989040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077373/real 1769077373] req@ffff969e0a7fed80 x1855007923854208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077373.989041:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923854720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077373.989119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923854208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077373.989130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077373.989135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077373.989139:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923854208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077373.989163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077373.989180:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077373.989187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077373/real 1769077373] req@ffff969e0a7fe680 x1855007923854336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077373.989214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923854336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077373.989220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077373.989222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077373.989224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077373.989228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077373.989242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077373.989248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077373/real 1769077373] req@ffff969e0a7fce00 x1855007923854464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077373.989275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923854464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077373.989281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077373.989283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077373.989285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077373.989290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077373.989305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077373.989312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077373/real 1769077373] req@ffff969e0a7fc700 x1855007923854592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077373.989319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923854592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077373.989334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077373.989337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077373.989339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077373.989344:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077373.989357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077373.989364:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077373/real 1769077373] req@ffff969e0a7fea00 x1855007923854720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077373.989376:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923854720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077373.989385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077373.989389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077373.989392:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077374.308706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077375.012791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077375.332698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077376.036767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077376.356682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077377.060800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077377.380709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077378.084812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077378.404713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077379.108697:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077379.108709:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077379.108714:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077379.108726:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4068 00010000:00010000:3.0:1769077379.108732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077379.108738:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077379.108766:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077379.108803:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923854848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077379.108828:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077379.108831:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077379.108832:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077379.108835:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4068 00000100:00080000:0.0:1769077379.108838:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077379.108841:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077379.108858:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077379.108870:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923854976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077379.108876:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923854848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077379.108881:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077379.108888:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077379.108889:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077379.108893:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4068 00000100:00080000:0.0:1769077379.108895:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077379.108900:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077379.108907:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077379.108935:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923855104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077379.108947:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077379.108951:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1769077379.108956:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077379.108962:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077379.108967:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4068 00000100:00080000:0.0:1769077379.108973:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077379.108977:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077379.108984:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077379.108995:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077379/real 1769077379] req@ffff969e06a60000 x1855007923854848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077379.108999:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923855232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077379.109029:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077379.109037:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077379.109040:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077379.109044:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4068 00000100:00080000:0.0:1769077379.109047:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077379.109051:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077379.109054:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077379.109057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923854848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077379.109065:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923855360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077379.109116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077379.109123:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077379.109128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923854848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077379.109181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077379.109209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077379.109220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077379/real 1769077379] req@ffff969e06a60a80 x1855007923854976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077379.109229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923854976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077379.109235:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077379.109238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077379.109240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923854976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077379.109244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077379.109256:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077379.109262:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077379/real 1769077379] req@ffff969e06a63800 x1855007923855104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077379.109269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923855104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077379.109274:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077379.109276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077379.109278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077379.109282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077379.109293:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077379.109299:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077379/real 1769077379] req@ffff969e06a60700 x1855007923855232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077379.109313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923855232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077379.109319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077379.109321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077379.109323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077379.109327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077379.109339:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077379.109345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077379/real 1769077379] req@ffff969e06a62d80 x1855007923855360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077379.109352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923855360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077379.109358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077379.109360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077379.109362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077379.428699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077380.132795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077380.452669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077381.156770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077381.476694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077382.180807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077382.500696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077383.204728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077383.524723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077384.228691:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077384.228702:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077384.228707:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077384.228718:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4073 00000100:00080000:0.0:1769077384.228751:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769077384.228755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077384.228759:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077384.228798:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923855488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077384.228822:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077384.228825:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077384.228826:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077384.228830:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4073 00000100:00080000:0.0:1769077384.228832:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077384.228835:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077384.228844:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077384.228853:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923855616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077384.228864:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077384.228867:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923855488:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077384.228869:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077384.228873:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077384.228881:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4073 00000100:00080000:0.0:1769077384.228885:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077384.228889:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077384.228895:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077384.228910:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923855744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077384.228921:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077384.228931:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077384.228932:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077384.228936:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4073 00000100:00080000:0.0:1769077384.228940:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077384.228944:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077384.228948:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077384.228955:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077384.228960:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923855872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077384.228969:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077384.228972:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077384.228974:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077384.228976:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4073 00000100:00080000:0.0:1769077384.228979:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077384.228982:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077384.228985:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077384.228993:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923856000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077384.229002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077384/real 1769077384] req@ffff969e06a61180 x1855007923855488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077384.229025:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923855488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077384.229051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077384.229057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077384.229061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923855488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077384.229123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077384.229149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077384.229162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077384/real 1769077384] req@ffff969e06a62680 x1855007923855616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077384.229176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923855616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077384.229184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077384.229188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077384.229192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077384.229199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077384.229217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077384.229227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077384/real 1769077384] req@ffff969e06a61f80 x1855007923855744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077384.229239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923855744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077384.229247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077384.229251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077384.229254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077384.229268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077384.229289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077384.229300:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077384/real 1769077384] req@ffff969e06a60380 x1855007923855872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077384.229314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923855872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077384.229323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077384.229328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077384.229331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923855872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077384.229338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077384.229357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077384.229368:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077384/real 1769077384] req@ffff969e06a61500 x1855007923856000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077384.229380:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923856000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077384.229389:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077384.229393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077384.229396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077384.548733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077385.252757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077385.572698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077386.276781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077386.596734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077387.300803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077387.620767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077388.324792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077388.644735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077389.348724:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077389.348736:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077389.348741:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077389.348751:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4078 00000100:00080000:3.0:1769077389.348770:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077389.348779:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077389.348782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077389.348832:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923856128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077389.348862:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077389.348865:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077389.348868:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077389.348873:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4078 00000100:00080000:3.0:1769077389.348878:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077389.348883:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077389.348893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923856128:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077389.348896:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077389.348944:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923856256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077389.348956:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077389.348960:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077389.348962:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077389.348965:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4078 00000100:00080000:3.0:1769077389.348970:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077389.348974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077389.348980:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077389.348986:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077389.349029:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077389/real 1769077389] req@ffff969e0a7fdf80 x1855007923856128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077389.349034:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923856384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077389.349060:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077389.349064:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077389.349066:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077389.349070:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4078 00000100:00100000:1.0:1769077389.349074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923856128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077389.349076:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077389.349083:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077389.349091:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077389.349092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077389.349100:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077389.349104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923856128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077389.349107:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923856512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077389.349116:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077389.349118:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077389.349120:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077389.349129:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4078 00000100:00100000:1.0:1769077389.349133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856256:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769077389.349134:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077389.349141:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077389.349146:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077389.349158:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923856640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077389.349166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077389.349182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077389/real 1769077389] req@ffff969e0a7fc380 x1855007923856256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077389.349193:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923856256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077389.349199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077389.349202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077389.349204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077389.349208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077389.349220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077389.349226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077389/real 1769077389] req@ffff969e0a7fd180 x1855007923856384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077389.349238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923856384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077389.349260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077389.349264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077389.349267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077389.349274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077389.349297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077389.349305:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077389/real 1769077389] req@ffff969e0a7fe300 x1855007923856512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077389.349313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923856512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077389.349319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077389.349321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077389.349323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077389.349335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077389.349347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077389.349352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077389/real 1769077389] req@ffff969e0a7fca80 x1855007923856640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077389.349360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923856640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077389.349365:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077389.349367:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077389.349374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077389.668759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077390.372801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077390.692725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077391.397148:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077391.716733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077392.420804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077392.740708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077393.444855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077393.764698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077394.468844:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077394.469057:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077394.469068:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077394.469074:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077394.469093:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4083 00000100:00080000:1.0:1769077394.469112:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077394.469123:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077394.469175:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923856768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077394.469207:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077394.469212:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077394.469214:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077394.469219:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4083 00000100:00080000:1.0:1769077394.469224:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077394.469229:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077394.469241:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077394.469257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923856768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077394.469260:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923856896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077394.469276:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077394.469279:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077394.469281:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077394.469291:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4083 00000100:00080000:1.0:1769077394.469308:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077394.469326:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077394.469332:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077394.469349:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923857024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077394.469360:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077394.469364:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077394.469366:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077394.469371:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4083 00000800:00000100:2.0:1769077394.469377:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769077394.469378:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077394.469391:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077394.469397:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077394.469412:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923857152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077394.469422:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077394/real 1769077394] req@ffff969e10b60700 x1855007923856768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077394.469428:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077394.469437:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077394.469442:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077394.469447:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4083 00000100:00080000:1.0:1769077394.469452:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077394.469455:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923856768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077394.469460:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077394.469471:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769077394.469475:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077394.469481:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077394.469488:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923857280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077394.469489:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923856768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077394.469542:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077394.469574:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077394.469629:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077394/real 1769077394] req@ffff969e10b62d80 x1855007923856896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077394.469647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923856896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077394.469658:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077394.469663:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077394.469667:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923856896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077394.469675:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077394.469698:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077394.469715:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077394/real 1769077394] req@ffff969e10b60000 x1855007923857024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077394.469745:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923857024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077394.469752:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077394.469755:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077394.469758:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077394.469763:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077394.469781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077394.469789:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077394/real 1769077394] req@ffff969e10b61180 x1855007923857152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077394.469797:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923857152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077394.469803:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077394.469806:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077394.469808:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077394.469812:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077394.469822:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077394.469827:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077394/real 1769077394] req@ffff969e10b60380 x1855007923857280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077394.469835:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923857280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077394.469850:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077394.469853:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077394.469855:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077394.788720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077395.492864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077395.812716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077396.517294:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077396.836711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077397.540890:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077397.860662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077398.564779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077398.884730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077399.588776:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077399.588789:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077399.588794:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077399.588806:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4088 00000100:00080000:2.0:1769077399.588818:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077399.588842:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077399.588878:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923857408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769077399.588890:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077399.588910:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077399.588915:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077399.588916:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077399.588920:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4088 00000100:00080000:2.0:1769077399.588923:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077399.588926:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077399.588934:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077399.588944:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923857536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077399.588962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923857408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077399.588970:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077399.588980:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077399.588982:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077399.588988:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4088 00000100:00080000:2.0:1769077399.588993:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077399.588998:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077399.589004:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077399.589021:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923857664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077399.589033:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077399.589037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077399.589043:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077399.589046:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077399.589051:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4088 00000100:00080000:2.0:1769077399.589058:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077399.589062:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077399.589067:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077399.589077:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923857792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077399.589085:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077399/real 1769077399] req@ffff969e0a46ed80 x1855007923857408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077399.589086:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077399.589096:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077399.589099:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077399.589106:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4088 00000100:00100000:1.0:1769077399.589113:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923857408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077399.589116:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077399.589125:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077399.589128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769077399.589130:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077399.589135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077399.589143:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923857408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077399.589144:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923857920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077399.589180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077399.589220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077399.589235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077399/real 1769077399] req@ffff969e0a46e300 x1855007923857536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077399.589250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923857536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077399.589260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077399.589264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077399.589268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077399.589275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077399.589297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077399.589310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077399/real 1769077399] req@ffff969e0a46f100 x1855007923857664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077399.589337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923857664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077399.589343:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077399.589346:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077399.589348:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077399.589353:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077399.589370:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077399.589377:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077399/real 1769077399] req@ffff969e0a46df80 x1855007923857792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077399.589384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923857792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077399.589390:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077399.589393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077399.589394:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077399.589399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077399.589417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077399.589428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077399/real 1769077399] req@ffff969e0a46ca80 x1855007923857920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077399.589440:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923857920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077399.589455:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077399.589458:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077399.589460:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923857920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077399.908668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077400.612773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077400.932665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077401.636870:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077401.956756:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077402.660734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077402.980670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077403.684772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077404.004736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077404.708673:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077404.708687:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077404.708692:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077404.708719:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4093 00000100:00080000:2.0:1769077404.708731:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077404.708754:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077404.708773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077404.708801:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923858048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077404.708824:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077404.708827:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077404.708828:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077404.708831:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4093 00000100:00080000:2.0:1769077404.708835:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077404.708838:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077404.708847:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077404.708858:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923858176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077404.708866:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077404.708878:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923858048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077404.708889:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077404.708892:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077404.708897:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4093 00000100:00080000:2.0:1769077404.708902:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077404.708907:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077404.708913:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077404.708929:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923858304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077404.708941:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077404.708945:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077404.708947:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077404.708951:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4093 00000100:00080000:2.0:1769077404.708956:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077404.708960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077404.708965:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077404.708971:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077404.708984:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923858432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077404.708995:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077404.708999:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077404.709001:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077404.709005:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077404/real 1769077404] req@ffff969e0a46ea00 x1855007923858048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077404.709008:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4093 00000100:00080000:2.0:1769077404.709017:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077404.709024:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077404.709038:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077404.709041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923858048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077404.709051:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923858560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077404.709060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077404.709068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077404.709075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923858048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077404.709099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077404.709123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077404.709136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077404/real 1769077404] req@ffff969e0a46ce00 x1855007923858176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077404.709149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923858176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077404.709158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077404.709162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077404.709165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077404.709181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077404.709203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077404.709231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077404/real 1769077404] req@ffff969e0a46f800 x1855007923858304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077404.709244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923858304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077404.709252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077404.709256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077404.709259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077404.709266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077404.709310:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077404.709327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077404/real 1769077404] req@ffff969e0a46d880 x1855007923858432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077404.709340:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923858432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077404.709348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077404.709352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077404.709355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077404.709361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077404.709379:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077404.709390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077404/real 1769077404] req@ffff969e0a46c380 x1855007923858560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077404.709413:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923858560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077404.709422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077404.709425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077404.709428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077405.028745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077405.732781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077406.052744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077406.756808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077407.076760:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077407.780832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077408.100817:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077408.804930:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077409.124802:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077409.828769:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077409.828781:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1769077409.828804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077409.828808:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077409.828821:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4099 00000100:00080000:0.0:1769077409.828834:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077409.828841:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077409.828877:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923858688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077409.828928:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077409.828933:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077409.828935:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077409.828940:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4099 00000100:00080000:0.0:1769077409.828945:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077409.828950:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077409.828973:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077409.828982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923858688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077409.829002:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923858816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077409.829024:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077409.829030:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077409.829034:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077409.829041:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4099 00000100:00080000:0.0:1769077409.829064:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077409.829072:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077409.829083:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077409.829083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077409.829112:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923858944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077409.829129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077409/real 1769077409] req@ffff969e06a63100 x1855007923858688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077409.829173:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077409.829179:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077409.829184:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077409.829185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923858688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077409.829197:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4099 00000100:00080000:0.0:1769077409.829210:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077409.829214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077409.829220:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077409.829227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:0.0:1769077409.829231:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077409.829234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923858688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077409.829254:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923859072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077409.829270:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077409.829274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858816:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769077409.829276:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077409.829284:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077409.829307:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4099 00000100:00080000:0.0:1769077409.829312:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077409.829317:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077409.829323:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077409.829331:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077409.829342:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923859200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077409.829352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077409/real 1769077409] req@ffff969e06a61880 x1855007923858816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077409.829372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923858816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077409.829400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077409.829405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077409.829409:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077409.829417:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077409.829439:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077409.829450:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077409/real 1769077409] req@ffff969e06a61500 x1855007923858944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077409.829461:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923858944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077409.829469:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077409.829473:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077409.829476:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923858944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077409.829493:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077409.829512:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077409.829521:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077409/real 1769077409] req@ffff969e06a60380 x1855007923859072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077409.829532:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923859072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077409.829540:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077409.829543:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077409.829554:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077409.829578:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077409.829597:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077409.829606:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077409/real 1769077409] req@ffff969e06a61f80 x1855007923859200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077409.829616:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923859200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077409.829667:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077409.829671:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077409.829674:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077410.148747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077410.852827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077411.172778:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077411.876792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077412.196725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077412.900826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077413.220724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077413.924772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077414.244704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077414.948709:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077414.948721:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077414.948725:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077414.948736:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4104 00000100:00080000:0.0:1769077414.948748:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077414.948756:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077414.948777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077414.948808:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923859328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077414.948832:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077414.948835:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077414.948837:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077414.948840:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4104 00000100:00080000:0.0:1769077414.948842:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077414.948845:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077414.948855:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077414.948865:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923859456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077414.948873:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077414.948877:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077414.948879:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077414.948880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923859328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077414.948888:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4104 00000100:00080000:0.0:1769077414.948892:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077414.948894:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077414.948898:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077414.948928:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923859584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077414.948937:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077414.948939:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077414.948941:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077414.948944:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4104 00000100:00080000:0.0:1769077414.948948:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769077414.948960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077414.948962:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077414.948970:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077414.948988:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923859712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077414.949000:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077414.949001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077414/real 1769077414] req@ffff969e06a62680 x1855007923859328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077414.949010:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077414.949015:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077414.949022:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4104 00000100:00100000:1.0:1769077414.949027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923859328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077414.949028:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077414.949037:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077414.949045:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077414.949048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077414.949054:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077414.949060:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923859840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077414.949060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923859328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077414.949092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077414.949121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077414.949149:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077414/real 1769077414] req@ffff969e06a61180 x1855007923859456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077414.949162:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923859456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077414.949169:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077414.949172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077414.949175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077414.949180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077414.949194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077414.949201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077414/real 1769077414] req@ffff969e06a62d80 x1855007923859584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077414.949209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923859584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077414.949214:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077414.949215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077414.949217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077414.949221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077414.949232:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077414.949238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077414/real 1769077414] req@ffff969e06a60700 x1855007923859712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077414.949245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923859712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077414.949257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077414.949259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077414.949262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077414.949266:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077414.949279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077414.949285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077414/real 1769077414] req@ffff969e06a63800 x1855007923859840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077414.949292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923859840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077414.949298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077414.949300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077414.949302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923859840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077415.268710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077415.972763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077416.292709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077416.996809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077417.316677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077418.020752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077418.340677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077419.044779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077419.364669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077420.068702:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077420.068715:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077420.068720:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077420.068733:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4109 00000100:00080000:3.0:1769077420.068747:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769077420.068752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077420.068761:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077420.068807:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923859968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077420.068830:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077420.068834:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077420.068836:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077420.068841:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4109 00000100:00080000:3.0:1769077420.068845:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077420.068849:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077420.068860:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077420.068870:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923859968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077420.068873:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923860096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077420.068882:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077420.068903:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077420.068905:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077420.068909:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4109 00000100:00080000:3.0:1769077420.068912:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077420.068916:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077420.068921:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077420.068931:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923860224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077420.068940:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077420.068943:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1769077420.068945:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077420.068945:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077420.068949:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4109 00000100:00080000:3.0:1769077420.068952:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077420.068956:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077420.068960:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077420.068971:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923860352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077420.068980:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077420/real 1769077420] req@ffff969e0a7fd880 x1855007923859968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077420.068980:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077420.068985:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077420.068987:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077420.068990:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4109 00000100:00080000:3.0:1769077420.068993:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077420.068997:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077420.069000:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923859968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077420.069007:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077420.069012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077420.069016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077420.069017:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923860480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077420.069021:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923859968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077420.069054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077420.069076:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077420.069087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077420/real 1769077420] req@ffff969e0a7ffb80 x1855007923860096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077420.069095:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923860096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077420.069101:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077420.069103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077420.069106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077420.069110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077420.069121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077420.069126:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077420/real 1769077420] req@ffff969e0a7fdc00 x1855007923860224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077420.069134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923860224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077420.069139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077420.069140:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077420.069142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077420.069146:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077420.069156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077420.069161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077420/real 1769077420] req@ffff969e0a7ff800 x1855007923860352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077420.069174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923860352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077420.069180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077420.069182:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077420.069184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077420.069187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077420.069199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077420.069206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077420/real 1769077420] req@ffff969e0a7fc000 x1855007923860480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077420.069219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923860480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077420.069228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077420.069231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077420.069234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077420.388713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077421.092743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077421.412711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077422.116743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077422.436740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077423.140789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077423.460704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077424.164785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077424.484691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077425.188738:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077425.188748:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077425.188752:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077425.188761:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4114 00010000:00010000:0.0:1769077425.188762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077425.188772:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077425.188779:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077425.188812:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923860608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077425.188835:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077425.188838:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077425.188854:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077425.188858:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4114 00000100:00080000:2.0:1769077425.188862:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077425.188867:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077425.188869:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923860608:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077425.188876:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077425.188887:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923860736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077425.188895:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077425.188898:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077425.188899:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077425.188902:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4114 00000100:00080000:2.0:1769077425.188905:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077425.188908:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077425.188910:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077425.188920:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923860864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077425.188926:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077425.188929:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077425.188931:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077425.188934:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4114 00000100:00080000:2.0:1769077425.188938:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077425.188940:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077425.188943:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077425.188946:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077425.188959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923860992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077425.188967:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077425.188970:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077425.188971:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077425.188975:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4114 00000100:00080000:2.0:1769077425.188978:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077425.188983:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077425/real 1769077425] req@ffff969e0a46d500 x1855007923860608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077425.188983:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077425.188991:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077425.189004:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923861120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077425.189009:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923860608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077425.189043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077425.189049:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077425.189054:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923860608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077425.189079:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077425.189099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077425.189108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077425/real 1769077425] req@ffff969e0a46f480 x1855007923860736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077425.189117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923860736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077425.189123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077425.189125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077425.189127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077425.189132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077425.189143:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077425.189157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077425/real 1769077425] req@ffff969e0a46c380 x1855007923860864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077425.189165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923860864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077425.189171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077425.189173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077425.189175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077425.189185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077425.189200:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077425.189206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077425/real 1769077425] req@ffff969e0a46d880 x1855007923860992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077425.189214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923860992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077425.189220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077425.189221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077425.189223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923860992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077425.189227:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077425.189236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077425.189242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077425/real 1769077425] req@ffff969e0a46f800 x1855007923861120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077425.189249:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923861120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077425.189254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077425.189256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077425.189258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077425.508718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077426.212788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077426.532695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077427.236774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077427.556770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077428.260849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077428.580707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077429.284806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077429.604750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077430.308740:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077430.308755:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077430.308759:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077430.308768:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4119 00000100:00080000:3.0:1769077430.308780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077430.308789:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077430.308790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077430.308826:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923861248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077430.308854:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077430.308857:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077430.308858:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077430.308861:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4119 00000100:00080000:3.0:1769077430.308864:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077430.308867:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077430.308876:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077430.308887:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923861376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077430.308897:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077430.308899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923861248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077430.308901:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077430.308905:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077430.308920:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4119 00000100:00080000:3.0:1769077430.308939:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077430.308945:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077430.308949:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077430.308959:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923861504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077430.308966:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077430.308969:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077430.308970:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077430.308973:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4119 00000100:00080000:3.0:1769077430.308975:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077430.308979:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077430.308986:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077430.309000:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923861632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077430.309008:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077430.309012:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077430.309013:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077430.309019:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4119 00000100:00080000:3.0:1769077430.309024:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077430.309029:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077430.309037:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077430.309046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077430.309053:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923861760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077430.309106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077430.309142:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077430.309169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077430.628725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077431.332784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077431.652753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077432.356774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077432.676717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077433.380809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077433.380877:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769077433.380921:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077433.380949:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077433.380985:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077433.380996:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077433.381006:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077433.381021:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077433.381061:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077430/real 1769077433] req@ffff969e0a7ffb80 x1855007923861760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077433.381110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923861760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077433.381127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077433.381140:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077433.381145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077433.381155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077430/real 1769077433] req@ffff969e0a7fc000 x1855007923861376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077433.381170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923861376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077433.381181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077433.381184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077433.381188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077433.381197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077430/real 1769077433] req@ffff969e0a7ff800 x1855007923861504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077433.381211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923861504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077433.381270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077433.381277:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077433.381281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077433.381290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077430/real 1769077433] req@ffff969e0a7fdc00 x1855007923861632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077433.381307:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923861632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077433.381316:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077433.381321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077433.381333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923861632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077433.381343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077430/real 1769077433] req@ffff969e0a7fd500 x1855007923861248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077433.381358:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923861248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077433.381368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077433.381372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077433.381376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923861248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077433.700693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077434.404790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077434.724704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077435.428776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077435.428812:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077435.428819:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077435.428824:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077435.428832:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4124 00000100:00080000:2.0:1769077435.428843:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077435.428851:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077435.428902:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923861888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077435.428933:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077435.428937:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077435.428940:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077435.428946:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4124 00000100:00080000:2.0:1769077435.428952:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077435.428959:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077435.428961:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923861888:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077435.428970:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077435.428987:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923862016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077435.428997:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077435.429000:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077435.429001:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077435.429005:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4124 00000100:00080000:2.0:1769077435.429008:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077435.429011:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077435.429014:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077435.429023:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923862144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077435.429031:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077435.429033:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077435.429035:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077435.429040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077435.429051:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4124 00000100:00080000:2.0:1769077435.429057:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077435.429062:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077435.429069:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077435.429084:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923862272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077435.429090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077435/real 1769077435] req@ffff969e0a46ce00 x1855007923861888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077435.429096:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077435.429101:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077435.429104:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077435.429113:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4124 00000100:00080000:2.0:1769077435.429119:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077435.429125:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077435.429130:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077435.429135:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923861888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077435.429146:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923862400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077435.429151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077435.429161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077435.429168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923861888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077435.429200:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077435.429246:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077435.429262:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077435/real 1769077435] req@ffff969e0a46ea00 x1855007923862016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077435.429275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923862016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077435.429283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077435.429286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077435.429290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077435.429297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077435.429316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077435.429328:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077435/real 1769077435] req@ffff969e0a46ca80 x1855007923862144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077435.429341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923862144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077435.429364:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077435.429369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077435.429372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077435.429380:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077435.429404:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077435.429416:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077435/real 1769077435] req@ffff969e0a46df80 x1855007923862272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077435.429429:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923862272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077435.429439:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077435.429443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077435.429447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077435.429453:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077435.429472:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077435.429483:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077435/real 1769077435] req@ffff969e0a46f100 x1855007923862400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077435.429497:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923862400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077435.429506:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077435.429509:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077435.429523:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077435.748734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077436.452776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077436.772685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077437.476740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077437.796695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077438.500781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077438.820668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077439.524776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077439.844699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077440.548730:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077440.548745:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077440.548752:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077440.548764:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4129 00000100:00080000:3.0:1769077440.548780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769077440.548785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077440.548790:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077440.548840:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923862528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077440.548869:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077440.548874:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077440.548876:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077440.548881:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4129 00000100:00080000:3.0:1769077440.548886:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077440.548891:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077440.548907:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077440.548912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923862528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077440.548925:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923862656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077440.548939:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077440.548943:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077440.548945:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077440.548950:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4129 00000100:00080000:3.0:1769077440.548954:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077440.548959:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077440.548981:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077440.548986:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077440.549001:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923862784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077440.549014:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077440.549019:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1769077440.549021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077440/real 1769077440] req@ffff969e0a7fd880 x1855007923862528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077440.549023:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077440.549034:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4129 00000100:00080000:3.0:1769077440.549041:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077440.549044:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923862528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077440.549049:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077440.549059:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769077440.549060:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077440.549068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077440.549074:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923862528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077440.549088:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923862912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077440.549103:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077440.549105:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862656:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769077440.549111:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077440.549115:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077440.549121:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4129 00000100:00080000:3.0:1769077440.549126:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077440.549131:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077440.549150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077440.549151:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077440.549164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077440/real 1769077440] req@ffff969e0a7fca80 x1855007923862656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077440.549175:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923863040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077440.549178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923862656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077440.549207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077440.549210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077440.549214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077440.549219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077440.549234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077440.549250:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077440/real 1769077440] req@ffff969e0a7fe300 x1855007923862784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077440.549259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923862784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077440.549265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077440.549267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077440.549270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077440.549280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077440.549292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077440.549298:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077440/real 1769077440] req@ffff969e0a7fd180 x1855007923862912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077440.549305:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923862912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077440.549311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077440.549312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077440.549314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923862912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077440.549318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077440.549327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077440.549333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077440/real 1769077440] req@ffff969e0a7fc380 x1855007923863040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077440.549340:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923863040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077440.549350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077440.549353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077440.549355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077440.868735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077441.572777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077441.892676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077442.596741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077442.916768:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077443.620718:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077443.940675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077444.644723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077444.964713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077445.668796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077445.668875:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077445.668895:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077445.668925:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077445.668937:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4134 00000100:00080000:3.0:1769077445.668957:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077445.668966:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077445.669011:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923863168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077445.669078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923863168:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769077445.669146:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077445.669176:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077445.669179:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077445.669181:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077445.669185:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4134 00000100:00080000:3.0:1769077445.669188:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077445.669191:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077445.669200:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077445.669221:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923863296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077445.669279:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077445.669282:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077445.669284:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077445.669287:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4134 00000100:00080000:3.0:1769077445.669296:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077445.669301:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077445.669305:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077445.669318:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923863424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077445.669341:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077445/real 1769077445] req@ffff969e0a7fdf80 x1855007923863168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077445.669381:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077445.669386:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077445.669388:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077445.669392:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4134 00000100:00080000:3.0:1769077445.669397:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077445.669401:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077445.669407:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923863168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077445.669409:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077445.669428:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923863552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077445.669430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077445.669448:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077445.669451:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077445.669452:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077445.669458:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077445.669472:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4134 00000100:00100000:1.0:1769077445.669473:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923863168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077445.669476:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077445.669490:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077445.669495:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077445.669506:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923863680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077445.669559:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077445.669669:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077445.669690:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077445/real 1769077445] req@ffff969e0a7fea00 x1855007923863296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077445.669706:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923863296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077445.669718:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077445.669723:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077445.669729:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077445.669754:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077445.669781:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077445.669796:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077445/real 1769077445] req@ffff969e0a7fc700 x1855007923863424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077445.669809:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923863424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077445.669818:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077445.669822:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077445.669826:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077445.669833:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077445.669854:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077445.669865:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077445/real 1769077445] req@ffff969e0a7fce00 x1855007923863552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077445.669878:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923863552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077445.669888:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077445.669892:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077445.669896:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077445.669902:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077445.669922:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077445.669934:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077445/real 1769077445] req@ffff969e0a7fe680 x1855007923863680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077445.669955:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923863680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077445.669965:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077445.669969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077445.669973:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077445.988711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077446.692707:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077447.012699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077447.716781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077448.036699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077448.740745:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077449.060754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077449.764836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077450.084710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077450.788743:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077450.788755:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077450.788760:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077450.788770:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4139 00000100:00080000:1.0:1769077450.788781:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077450.788787:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077450.788825:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077450.788836:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923863808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077450.788872:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077450.788875:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077450.788876:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077450.788879:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4139 00000100:00080000:1.0:1769077450.788883:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077450.788889:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077450.788899:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077450.788915:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923863936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077450.788917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923863808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077450.788996:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077450.788999:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077450.789001:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077450.789005:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4139 00000100:00080000:1.0:1769077450.789009:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077450.789013:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077450.789018:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077450.789023:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077450.789036:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923864064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077450.789047:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077450.789050:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077450.789051:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077450.789058:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4139 00000100:00080000:1.0:1769077450.789063:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077450.789067:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077450/real 1769077450] req@ffff969e10b60380 x1855007923863808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077450.789068:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077450.789079:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077450.789100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923863808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077450.789104:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923864192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077450.789136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077450.789139:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077450.789149:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077450.789149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:1.0:1769077450.789154:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077450.789162:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4139 00000100:00100000:2.0:1769077450.789165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923863808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077450.789178:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077450.789183:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077450.789188:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077450.789201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077450.789204:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923864320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077450.789241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077450.789254:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077450/real 1769077450] req@ffff969e10b61180 x1855007923863936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077450.789263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923863936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077450.789269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077450.789271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077450.789274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923863936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077450.789278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077450.789307:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077450.789314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077450/real 1769077450] req@ffff969e10b60000 x1855007923864064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077450.789321:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923864064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077450.789326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077450.789328:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077450.789330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077450.789334:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077450.789345:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077450.789350:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077450/real 1769077450] req@ffff969e10b62d80 x1855007923864192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077450.789357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923864192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077450.789363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077450.789365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077450.789367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077450.789378:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077450.789388:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077450.789394:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077450/real 1769077450] req@ffff969e10b60700 x1855007923864320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077450.789411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923864320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077450.789417:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077450.789419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077450.789421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077451.108702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077451.812879:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077452.132718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077452.836760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077453.156769:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077453.860782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077454.180706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077454.884758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077455.204713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077455.908741:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077455.908757:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077455.908762:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077455.908775:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4145 00000100:00080000:1.0:1769077455.908787:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077455.908795:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077455.908847:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923864448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077455.908914:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077455.908919:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077455.908921:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077455.908928:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4145 00000100:00080000:1.0:1769077455.908933:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077455.908937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923864448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077455.908942:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077455.908956:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077455.908973:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923864576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077455.908987:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077455.909014:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769077455.909016:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077455.909020:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077455.909025:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4145 00000100:00080000:1.0:1769077455.909031:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077455.909036:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077455.909042:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077455.909053:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923864704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077455.909061:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077455.909065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077455/real 1769077455] req@ffff969e10b62680 x1855007923864448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077455.909066:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077455.909071:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077455.909080:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4145 00000100:00080000:1.0:1769077455.909084:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077455.909087:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077455.909091:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077455.909092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923864448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077455.909103:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923864832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077455.909104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077455.909114:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077455.909118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:1.0:1769077455.909119:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077455.909123:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077455.909126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923864448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077455.909130:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4145 00000100:00080000:1.0:1769077455.909134:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077455.909137:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077455.909146:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077455.909156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077455.909157:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923864960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077455.909182:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077455.909195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077455/real 1769077455] req@ffff969e10b62a00 x1855007923864576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077455.909220:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923864576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077455.909228:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077455.909231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077455.909234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077455.909239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077455.909255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077455.909266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077455/real 1769077455] req@ffff969e10b60a80 x1855007923864704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769077455.909272:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077455.909283:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923864704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077455.909294:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077455.909298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077455.909302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077455.909309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077455.909339:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077455.909347:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077455/real 1769077455] req@ffff969e10b61f80 x1855007923864832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077455.909355:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923864832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077455.909371:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077455.909373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077455.909376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077455.909389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077455.909402:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077455.909409:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077455/real 1769077455] req@ffff969e10b61500 x1855007923864960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077455.909417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923864960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077455.909423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077455.909425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077455.909427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923864960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077456.228669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077456.932727:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077457.252749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077457.957155:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077458.276688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077458.981045:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077459.300712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077460.004862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077460.324703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077461.029022:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077461.029037:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077461.029043:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077461.029058:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4150 00000100:00080000:1.0:1769077461.029073:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077461.029093:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077461.029150:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923865088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077461.029189:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077461.029194:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077461.029197:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077461.029203:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4150 00000100:00080000:1.0:1769077461.029208:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077461.029213:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077461.029228:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077461.029246:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923865216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077461.029261:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077461.029272:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769077461.029273:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923865088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077461.029276:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077461.029294:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4150 00000100:00080000:1.0:1769077461.029322:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077461.029327:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077461.029336:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077461.029354:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923865344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077461.029367:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077461.029371:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077461.029375:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077461.029380:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4150 00000100:00080000:1.0:1769077461.029386:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077461.029392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769077461.029397:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077461.029406:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077461.029426:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923865472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077461.029439:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077461.029444:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077461.029448:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077461.029457:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4150 00000100:00100000:2.0:1769077461.029462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077461/real 1769077461] req@ffff969e10b61500 x1855007923865088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077461.029466:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077461.029475:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077461.029482:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077461.029504:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923865600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077461.029519:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923865088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077461.029568:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077461.029576:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077461.029584:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923865088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077461.029681:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077461.029728:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077461.029744:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077461/real 1769077461] req@ffff969e10b61f80 x1855007923865216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769077461.029747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077461.029759:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923865216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077461.029770:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077461.029774:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077461.029778:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077461.029785:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077461.029828:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077461.029839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077461/real 1769077461] req@ffff969e10b60a80 x1855007923865344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077461.029849:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923865344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077461.029857:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077461.029861:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077461.029865:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077461.029872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077461.029896:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077461.029923:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077461/real 1769077461] req@ffff969e10b62a00 x1855007923865472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077461.029945:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923865472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077461.029955:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077461.029958:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077461.029962:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077461.029969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077461.029993:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077461.030005:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077461/real 1769077461] req@ffff969e10b62680 x1855007923865600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077461.030018:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923865600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077461.030028:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077461.030032:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077461.030035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077461.348692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077462.053472:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077462.372753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077463.077093:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077463.396671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077464.100760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077464.420766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077465.124773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077465.444758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077466.148699:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077466.148722:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077466.148727:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077466.148735:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4155 00000100:00080000:1.0:1769077466.148745:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077466.148752:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077466.148784:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923865728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077466.148810:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077466.148814:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077466.148815:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077466.148818:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4155 00000100:00080000:1.0:1769077466.148820:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077466.148823:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077466.148830:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077466.148839:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923865856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077466.148846:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077466.148848:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077466.148850:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077466.148853:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4155 00000100:00080000:1.0:1769077466.148855:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077466.148857:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077466.148876:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077466.148888:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923865984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077466.148896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923865728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077466.148899:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077466.148906:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077466.148911:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077466.148918:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4155 00000100:00080000:1.0:1769077466.148923:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077466.148928:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077466.148934:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077466.148948:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923866112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077466.148959:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077466.148963:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077466.148965:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077466.148972:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4155 00000100:00080000:1.0:1769077466.148976:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077466.148979:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077466.148983:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077466.148995:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923866240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077466.149006:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077466.149066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077466/real 1769077466] req@ffff969e10b60700 x1855007923865728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1769077466.149092:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077466.149093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923865728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077466.149127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077466.149131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077466.149135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923865728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077466.149143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077466.149165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077466.149174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077466/real 1769077466] req@ffff969e10b62d80 x1855007923865856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077466.149183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923865856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077466.149189:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077466.149192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077466.149194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077466.149216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077466.149228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077466.149234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077466/real 1769077466] req@ffff969e10b60000 x1855007923865984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077466.149241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923865984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077466.149247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077466.149249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077466.149251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923865984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077466.149261:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077466.149275:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077466.149282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077466/real 1769077466] req@ffff969e10b61180 x1855007923866112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077466.149289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923866112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077466.149295:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077466.149297:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077466.149299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077466.149303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077466.149313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077466.149318:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077466/real 1769077466] req@ffff969e10b60380 x1855007923866240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077466.149325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923866240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077466.149333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077466.149336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077466.149339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077466.468747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077467.172810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077467.492657:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077468.196778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077468.516713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077469.220827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077469.540763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077470.244779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077470.564752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077471.268758:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077471.268772:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077471.268797:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077471.268809:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4160 00000100:00080000:1.0:1769077471.268823:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077471.268831:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077471.268876:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923866368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077471.268907:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077471.268910:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077471.268913:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077471.268917:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4160 00000100:00080000:1.0:1769077471.268921:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077471.268924:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077471.268936:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077471.268948:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923866496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077471.268959:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077471.268964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923866368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077471.268965:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077471.268969:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077471.268977:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4160 00000100:00080000:1.0:1769077471.268981:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077471.268984:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077471.268989:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077471.269000:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923866624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077471.269025:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077471.269029:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077471.269031:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077471.269036:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4160 00000100:00080000:1.0:1769077471.269040:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077471.269044:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769077471.269050:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:1.0:1769077471.269051:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077471.269065:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923866752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077471.269076:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077471.269080:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077471.269082:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077471.269086:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4160 00000100:00080000:1.0:1769077471.269090:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077471.269094:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077471.269098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077471/real 1769077471] req@ffff969e10b60380 x1855007923866368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769077471.269100:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077471.269111:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923866880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077471.269148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923866368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077471.269163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077471.269168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077471.269173:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923866368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077471.269205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077471.269231:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00010000:00010000:0.0:1769077471.269237:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077471.269246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077471/real 1769077471] req@ffff969e10b61180 x1855007923866496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077471.269261:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923866496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077471.269271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077471.269274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077471.269278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077471.269285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077471.269303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077471.269314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077471/real 1769077471] req@ffff969e10b60000 x1855007923866624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077471.269325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923866624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077471.269347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077471.269351:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077471.269355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077471.269362:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077471.269383:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077471.269393:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077471/real 1769077471] req@ffff969e10b62d80 x1855007923866752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077471.269405:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923866752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077471.269415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077471.269418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077471.269422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077471.269429:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077471.269447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077471.269457:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077471/real 1769077471] req@ffff969e10b60700 x1855007923866880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077471.269469:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923866880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077471.269478:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077471.269481:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077471.269491:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923866880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077471.588770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077472.292785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077472.612749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077473.316803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077473.636738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077474.340751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077474.660764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077475.364812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077475.684692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077476.388709:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077476.388725:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077476.388731:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769077476.388740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077476.388746:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4165 00000100:00080000:1.0:1769077476.388781:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077476.388788:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077476.388827:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923867008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077476.388852:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077476.388855:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077476.388856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077476.388859:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4165 00000100:00080000:1.0:1769077476.388862:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077476.388864:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077476.388873:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077476.388882:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923867136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077476.388891:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077476.388893:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077476.388894:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077476.388897:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4165 00000100:00080000:1.0:1769077476.388899:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077476.388903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923867008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077476.388904:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077476.388912:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077476.388919:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923867264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077476.388926:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077476.388934:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077476.388935:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077476.388939:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4165 00000100:00080000:1.0:1769077476.388942:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077476.388945:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077476.388949:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077476.388958:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923867392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077476.388964:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077476.388967:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077476.388969:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077476.388971:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4165 00000100:00080000:1.0:1769077476.388973:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077476.388975:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077476.388979:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077476.388983:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077476.388990:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923867520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077476.389020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077476/real 1769077476] req@ffff969e10b62680 x1855007923867008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077476.389035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923867008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077476.389044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077476.389048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077476.389051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923867008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077476.389074:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077476.389097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077476.389111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077476/real 1769077476] req@ffff969e10b62a00 x1855007923867136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077476.389125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923867136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077476.389134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077476.389137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077476.389140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077476.389145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077476.389161:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077476.389177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077476/real 1769077476] req@ffff969e10b60a80 x1855007923867264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077476.389185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923867264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077476.389191:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077476.389194:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077476.389196:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077476.389210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077476.389233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077476.389245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077476/real 1769077476] req@ffff969e10b61f80 x1855007923867392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077476.389259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923867392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077476.389268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077476.389272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077476.389275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077476.389280:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077476.389295:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077476.389302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077476/real 1769077476] req@ffff969e10b61500 x1855007923867520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077476.389310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923867520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077476.389323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077476.389325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077476.389328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077476.708704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077477.412791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077477.732664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077478.436759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077478.756738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077479.460813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077479.780672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077480.484726:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077480.804716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077481.508709:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077481.508723:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077481.508729:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077481.508742:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4170 00000100:00080000:3.0:1769077481.508776:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077481.508782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077481.508786:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077481.508830:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923867648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077481.508862:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077481.508866:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077481.508868:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077481.508873:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4170 00000100:00080000:3.0:1769077481.508877:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077481.508881:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077481.508891:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077481.508907:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923867776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077481.508920:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077481.508923:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077481.508926:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077481.508931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923867648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077481.508933:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4170 00000100:00080000:3.0:1769077481.508940:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077481.508945:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077481.508950:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077481.508963:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923867904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077481.508973:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077481.508983:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077481.508985:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077481.508989:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4170 00000100:00080000:3.0:1769077481.509002:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077481.509007:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769077481.509013:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077481.509013:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077481.509030:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923868032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077481.509042:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077481.509046:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077481.509048:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077481.509053:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4170 00000100:00100000:2.0:1769077481.509057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077481/real 1769077481] req@ffff969e0a7fed80 x1855007923867648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077481.509059:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077481.509068:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077481.509078:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077481.509088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923867648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077481.509092:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923868160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077481.509116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077481.509122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077481.509127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923867648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077481.509156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077481.509180:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077481.509191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077481/real 1769077481] req@ffff969e0a7fe680 x1855007923867776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077481.509203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923867776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077481.509212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077481.509215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077481.509219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077481.509233:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077481.509263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077481.509275:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077481/real 1769077481] req@ffff969e0a7fce00 x1855007923867904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077481.509288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923867904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077481.509297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077481.509301:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077481.509304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923867904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077481.509311:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077481.509338:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077481.509349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077481/real 1769077481] req@ffff969e0a7fc700 x1855007923868032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077481.509361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923868032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077481.509369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077481.509372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077481.509376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077481.509381:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077481.509397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077481.509406:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077481/real 1769077481] req@ffff969e0a7fea00 x1855007923868160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077481.509424:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923868160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077481.509433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077481.509436:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077481.509440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077481.828730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077482.532793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077482.852679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077483.556778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077483.876732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077484.580791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077484.900728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077485.604837:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077485.924708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077486.628699:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077486.628708:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077486.628712:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077486.628720:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4175 00000100:00080000:3.0:1769077486.628731:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077486.628737:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077486.628772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077486.628777:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923868288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077486.628813:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077486.628816:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077486.628817:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077486.628821:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4175 00000100:00080000:3.0:1769077486.628824:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077486.628827:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077486.628834:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077486.628844:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923868416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077486.628852:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077486.628855:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077486.628858:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077486.628859:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923868288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077486.628863:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4175 00000100:00080000:3.0:1769077486.628868:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077486.628871:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077486.628874:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077486.628882:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923868544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077486.628889:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077486.628891:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077486.628893:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077486.628895:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4175 00000100:00080000:3.0:1769077486.628917:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077486.628920:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077486.628924:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077486.628927:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077486.628938:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923868672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077486.628947:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077486.628949:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077486.628951:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077486.628954:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4175 00000100:00080000:3.0:1769077486.628958:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077486.628960:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077486.628963:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077486.628967:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077486/real 1769077486] req@ffff969e0a7fdf80 x1855007923868288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077486.628973:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923868800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077486.629010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923868288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077486.629020:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077486.629024:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077486.629028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923868288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077486.629048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077486.629067:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077486.629075:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077486/real 1769077486] req@ffff969e0a7fc380 x1855007923868416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077486.629083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923868416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077486.629100:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077486.629103:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077486.629105:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077486.629110:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077486.629134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077486.629141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077486/real 1769077486] req@ffff969e0a7fd180 x1855007923868544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077486.629149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923868544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077486.629155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077486.629157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077486.629160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077486.629163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077486.629179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077486.629187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077486/real 1769077486] req@ffff969e0a7fe300 x1855007923868672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077486.629195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923868672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077486.629200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077486.629202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077486.629209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077486.629214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077486.629228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077486.629234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077486/real 1769077486] req@ffff969e0a7fca80 x1855007923868800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077486.629243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923868800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077486.629249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077486.629251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077486.629252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923868800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077486.948754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077487.652795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077487.972713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077488.676795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077488.996691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077489.700762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077490.020794:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077490.724814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077491.044689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077491.748744:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077491.748758:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077491.748764:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077491.748777:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4180 00000100:00080000:2.0:1769077491.748790:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077491.748803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077491.748805:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077491.748855:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923868928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077491.748892:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077491.748897:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077491.748899:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077491.748904:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4180 00000100:00080000:2.0:1769077491.748908:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077491.748938:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077491.748943:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923868928:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077491.748953:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077491.748973:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923869056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077491.748985:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077491.748989:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077491.748992:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077491.748996:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4180 00000100:00080000:2.0:1769077491.749002:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077491.749007:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077491.749012:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077491.749024:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923869184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077491.749036:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077491.749038:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077491.749044:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077491.749046:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077491.749052:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4180 00000100:00080000:2.0:1769077491.749056:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077491.749061:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077491.749067:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077491.749081:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923869312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077491.749092:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077491/real 1769077491] req@ffff969e0a46e300 x1855007923868928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077491.749095:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077491.749108:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077491.749111:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077491.749117:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4180 00000100:00100000:1.0:1769077491.749120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923868928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077491.749124:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077491.749134:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077491.749140:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077491.749156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077491.749162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077491.749163:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923869440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077491.749167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923868928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077491.749196:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077491.749217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077491.749226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077491/real 1769077491] req@ffff969e0a46ed80 x1855007923869056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077491.749235:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923869056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077491.749241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077491.749244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077491.749246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077491.749250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077491.749261:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077491.749267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077491/real 1769077491] req@ffff969e0a46f100 x1855007923869184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077491.749304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923869184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077491.749311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077491.749313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077491.749315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077491.749325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077491.749342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077491.749349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077491/real 1769077491] req@ffff969e0a46df80 x1855007923869312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077491.749357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923869312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077491.749363:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077491.749366:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077491.749368:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077491.749372:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077491.749382:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077491.749387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077491/real 1769077491] req@ffff969e0a46ca80 x1855007923869440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077491.749394:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923869440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077491.749400:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077491.749401:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077491.749403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077492.068759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077492.772880:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077493.092735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077493.796782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077494.116744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077494.820838:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077495.140676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077495.844764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077496.164750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077496.868711:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077496.868720:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077496.868725:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077496.868734:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4186 00000100:00080000:2.0:1769077496.868745:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077496.868752:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077496.868788:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923869568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077496.868812:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077496.868815:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:0.0:1769077496.868818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077496.868818:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077496.868826:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4186 00000100:00080000:2.0:1769077496.868833:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077496.868837:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077496.868846:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077496.868865:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923869696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077496.868872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923869568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077496.868892:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077496.868894:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077496.868895:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077496.868900:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4186 00000100:00080000:2.0:1769077496.868904:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077496.868907:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077496.868911:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077496.868920:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923869824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077496.868926:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077496.868929:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077496.868930:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077496.868933:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4186 00000100:00080000:2.0:1769077496.868937:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077496.868939:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077496.868942:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077496.868951:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923869952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077496.868957:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077496.868959:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077496.868960:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077496.868963:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4186 00000100:00080000:2.0:1769077496.868970:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077496.868974:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077496.868977:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077496.868985:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923870080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077496.869024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077496.869075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077496.869108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077496.869132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077497.188670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077497.892787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077498.212720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077498.916777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077499.236806:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077499.940820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077499.940992:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769077499.941050:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077499.941072:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077499.941128:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077499.941154:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077499.941167:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077499.941179:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077499.941186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077496/real 1769077499] req@ffff969e0a46ce00 x1855007923869696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077499.941219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923869696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077499.941245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077499.941252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077499.941256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077499.941265:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077496/real 1769077499] req@ffff969e0a46f800 x1855007923869824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077499.941281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923869824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077499.941290:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077499.941295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077499.941299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077499.941307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077496/real 1769077499] req@ffff969e0a46d880 x1855007923869952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077499.941322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923869952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077499.941354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077499.941359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077499.941363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923869952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077499.941372:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077496/real 1769077499] req@ffff969e0a46c380 x1855007923870080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077499.941388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923870080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077499.941397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077499.941402:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077499.941405:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077499.941413:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077496/real 1769077499] req@ffff969e0a46ea00 x1855007923869568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077499.941427:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923869568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077499.941436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077499.941440:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077499.941444:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923869568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077500.260693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077500.964823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077501.284720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077501.988743:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077501.988757:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077501.988762:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077501.988773:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4191 00010000:00010000:0.0:1769077501.988786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077501.988786:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077501.988797:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077501.988834:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923870208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077501.988865:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077501.988885:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077501.988888:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077501.988894:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4191 00000100:00080000:3.0:1769077501.988899:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077501.988904:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077501.988910:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923870208:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077501.988917:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077501.988935:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923870336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077501.988948:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077501.988952:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077501.988954:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077501.988959:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4191 00000100:00080000:3.0:1769077501.988972:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077501.988976:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077501.988982:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077501.988985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077501.988999:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923870464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077501.989010:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077501.989014:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077501.989017:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077501.989022:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4191 00000100:00080000:3.0:1769077501.989026:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077501.989030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077501/real 1769077501] req@ffff969e0a7fd880 x1855007923870208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077501.989035:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077501.989050:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077501.989061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923870208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077501.989065:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923870592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077501.989085:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077501.989089:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077501.989094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077501.989096:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769077501.989119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923870208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077501.989135:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077501.989142:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4191 00000100:00080000:3.0:1769077501.989147:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077501.989151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870336:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769077501.989157:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077501.989165:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077501.989181:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923870720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077501.989192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077501.989221:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077501/real 1769077501] req@ffff969e0a7fd500 x1855007923870336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077501.989237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923870336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077501.989247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077501.989251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077501.989256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077501.989263:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077501.989285:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077501.989297:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077501/real 1769077501] req@ffff969e0a7fdc00 x1855007923870464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077501.989310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923870464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077501.989318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077501.989327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077501.989331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077501.989338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077501.989360:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077501.989372:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077501/real 1769077501] req@ffff969e0a7ff800 x1855007923870592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077501.989384:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923870592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077501.989399:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077501.989404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077501.989408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077501.989425:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077501.989449:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077501.989461:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077501/real 1769077501] req@ffff969e0a7fc000 x1855007923870720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077501.989473:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923870720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077501.989483:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077501.989487:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077501.989490:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077502.308691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077503.012793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077503.332729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077504.036775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077504.356671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077505.060831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077505.380648:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077506.084770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077506.404721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077507.108703:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077507.108718:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077507.108725:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077507.108739:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4196 00000100:00080000:3.0:1769077507.108753:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077507.108763:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077507.108772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077507.108813:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923870848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077507.108849:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077507.108854:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077507.108856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077507.108861:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4196 00000100:00080000:3.0:1769077507.108885:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077507.108890:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077507.108902:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077507.108903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923870848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077507.108922:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923870976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077507.108935:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077507.108939:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077507.108941:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077507.108945:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4196 00000100:00080000:3.0:1769077507.108948:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077507.108951:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077507.108955:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077507.108965:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923871104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077507.108972:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077507.108975:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077507.108976:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077507.108979:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4196 00000100:00080000:3.0:1769077507.108982:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077507.108987:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769077507.108988:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077507.108991:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077507.109005:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923871232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077507.109041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077507/real 1769077507] req@ffff969e0a7ffb80 x1855007923870848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077507.109047:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077507.109056:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077507.109060:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077507.109067:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4196 00000100:00080000:3.0:1769077507.109073:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077507.109077:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923870848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077507.109081:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077507.109091:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769077507.109097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077507.109104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077507.109111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923870848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077507.109111:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923871360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077507.109157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077507.109179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077507.109189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077507/real 1769077507] req@ffff969e0a7fc000 x1855007923870976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077507.109199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923870976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077507.109206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077507.109209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077507.109213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923870976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077507.109219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077507.109241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077507.109253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077507/real 1769077507] req@ffff969e0a7ff800 x1855007923871104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077507.109267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923871104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077507.109276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077507.109279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077507.109283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077507.109289:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077507.109313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077507.109324:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077507/real 1769077507] req@ffff969e0a7fdc00 x1855007923871232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077507.109347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923871232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077507.109357:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077507.109361:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077507.109365:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077507.109371:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077507.109395:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077507.109407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077507/real 1769077507] req@ffff969e0a7fd500 x1855007923871360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077507.109421:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923871360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077507.109430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077507.109434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077507.109438:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077507.428715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077508.132798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077508.452692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077509.156770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077509.476725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077510.180765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077510.500685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077511.204827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077511.524690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077512.228700:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077512.228715:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077512.228721:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077512.228735:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4201 00000100:00080000:3.0:1769077512.228750:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077512.228759:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077512.228788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077512.228811:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923871488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077512.228847:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077512.228851:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077512.228853:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077512.228858:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4201 00000100:00080000:3.0:1769077512.228863:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077512.228868:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077512.228894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923871488:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077512.228897:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077512.228913:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923871616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077512.228926:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077512.228942:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077512.228945:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077512.228950:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4201 00000100:00080000:3.0:1769077512.228976:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077512.228982:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769077512.228988:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077512.228988:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077512.229004:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923871744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077512.229012:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077512.229015:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077512.229017:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077512.229021:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4201 00000100:00080000:3.0:1769077512.229025:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077512.229027:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077512.229031:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077512.229032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077512/real 1769077512] req@ffff969e0a7fd880 x1855007923871488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077512.229051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923871488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077512.229054:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923871872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077512.229072:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077512.229074:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077512.229081:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077512.229085:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4201 00000100:00080000:3.0:1769077512.229088:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077512.229095:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077512.229103:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769077512.229105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077512.229111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077512.229116:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923872000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077512.229119:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923871488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077512.229150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077512.229171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077512.229180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077512/real 1769077512] req@ffff969e0a7fca80 x1855007923871616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077512.229189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923871616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077512.229195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077512.229198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077512.229200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077512.229204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077512.229215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077512.229220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077512/real 1769077512] req@ffff969e0a7fe300 x1855007923871744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077512.229228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923871744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077512.229233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077512.229235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077512.229237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077512.229246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077512.229266:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077512.229278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077512/real 1769077512] req@ffff969e0a7fd180 x1855007923871872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077512.229290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923871872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077512.229300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077512.229304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077512.229307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923871872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077512.229314:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077512.229335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077512.229346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077512/real 1769077512] req@ffff969e0a7fc380 x1855007923872000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077512.229357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923872000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077512.229364:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077512.229366:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077512.229369:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077512.548688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077513.252782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077513.572728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077514.276800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077514.596695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077515.300742:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077515.620713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077516.324841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077516.644689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077517.348727:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077517.348745:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077517.348750:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077517.348785:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4206 00000100:00080000:2.0:1769077517.348799:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077517.348805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077517.348811:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077517.348871:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923872128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077517.348905:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077517.348909:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077517.348911:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077517.348916:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4206 00000100:00080000:2.0:1769077517.348921:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077517.348927:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077517.348939:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077517.348954:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923872128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077517.348958:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923872256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077517.348977:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077517.348981:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077517.348984:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077517.348991:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4206 00000100:00080000:2.0:1769077517.348996:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077517.349001:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077517.349009:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077517.349026:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923872384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077517.349034:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077517.349065:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077517.349073:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077517/real 1769077517] req@ffff969e0a46f480 x1855007923872128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077517.349074:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077517.349078:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077517.349090:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4206 00000100:00080000:2.0:1769077517.349101:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077517.349107:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077517.349115:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077517.349135:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923872512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077517.349143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923872128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077517.349178:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077517.349179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077517.349189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769077517.349189:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077517.349198:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077517.349200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923872128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077517.349219:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4206 00000100:00080000:2.0:1769077517.349225:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077517.349229:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077517.349238:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077517.349239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077517.349271:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923872640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077517.349279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077517.349295:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077517/real 1769077517] req@ffff969e0a46d500 x1855007923872256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077517.349308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923872256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077517.349315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077517.349320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077517.349323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077517.349329:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077517.349340:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077517.349346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077517/real 1769077517] req@ffff969e0a46ea00 x1855007923872384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077517.349354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923872384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077517.349369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077517.349371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077517.349374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077517.349378:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077517.349392:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077517.349398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077517/real 1769077517] req@ffff969e0a46c380 x1855007923872512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077517.349406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923872512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077517.349411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077517.349413:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077517.349415:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077517.349428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077517.349438:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077517.349444:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077517/real 1769077517] req@ffff969e0a46d880 x1855007923872640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077517.349452:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923872640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077517.349457:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077517.349459:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077517.349467:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077517.668696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077518.372797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077518.692729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077519.396733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077519.716672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077520.420784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077520.740768:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077521.444847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077521.764742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077522.468714:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077522.468725:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077522.468730:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077522.468740:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4211 00010000:00010000:0.0:1769077522.468744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077522.468753:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077522.468816:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077522.468860:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923872768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077522.468889:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077522.468892:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077522.468894:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077522.468897:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4211 00000100:00080000:3.0:1769077522.468901:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077522.468904:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077522.468913:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077522.468923:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923872896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077522.468941:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923872768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077522.468965:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077522.468969:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077522.468972:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077522.468977:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4211 00000100:00080000:3.0:1769077522.468981:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077522.468984:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077522.468988:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077522.468999:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923873024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077522.469005:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077522.469009:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077522.469010:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077522.469013:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4211 00000100:00080000:3.0:1769077522.469016:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077522.469018:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077522.469022:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077522.469029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077522.469038:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923873152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077522.469047:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077522.469049:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077522.469051:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077522.469054:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4211 00000100:00080000:3.0:1769077522.469063:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077522.469067:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077522.469073:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077522.469074:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077522/real 1769077522] req@ffff969e0a7fdf80 x1855007923872768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077522.469087:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923873280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077522.469122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923872768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077522.469140:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077522.469144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077522.469147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923872768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077522.469171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077522.469191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077522.469200:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077522/real 1769077522] req@ffff969e0a7fea00 x1855007923872896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077522.469209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923872896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077522.469215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077522.469217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077522.469219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923872896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077522.469224:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077522.469234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077522.469256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077522/real 1769077522] req@ffff969e0a7fc700 x1855007923873024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077522.469264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923873024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077522.469270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077522.469272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077522.469274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077522.469278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077522.469290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077522.469296:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077522/real 1769077522] req@ffff969e0a7fce00 x1855007923873152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077522.469303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923873152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077522.469308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077522.469310:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077522.469312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077522.469316:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077522.469325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077522.469331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077522/real 1769077522] req@ffff969e0a7fe680 x1855007923873280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077522.469338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923873280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077522.469348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077522.469351:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077522.469353:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077522.788741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077523.492796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077523.812669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077524.517441:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077524.836724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077525.540927:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077525.860689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077526.564864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077526.884785:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077527.589002:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077527.589386:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077527.589397:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077527.589402:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077527.589414:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4216 00000100:00080000:2.0:1769077527.589435:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077527.589445:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077527.589501:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923873408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077527.589544:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077527.589548:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077527.589551:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077527.589557:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4216 00000100:00080000:2.0:1769077527.589562:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077527.589567:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077527.589582:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077527.589611:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923873408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077527.589637:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923873536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077527.589649:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077527.589651:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077527.589654:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077527.589658:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4216 00000100:00080000:2.0:1769077527.589663:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077527.589667:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077527.589685:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077527.589698:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923873664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077527.589709:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077527.589723:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077527.589726:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077527.589731:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4216 00000800:00000100:1.0:1769077527.589736:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077527.589740:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077527.589748:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077527.589754:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077527.589778:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923873792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077527.589784:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077527/real 1769077527] req@ffff969e0a46f800 x1855007923873408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077527.589789:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077527.589797:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077527.589801:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077527.589808:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4216 00000100:00080000:2.0:1769077527.589812:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077527.589815:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077527.589819:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077527.589824:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923873408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077527.589831:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923873920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077527.589854:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077527.589863:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077527.589869:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923873408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077527.589898:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077527.589925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077527.589945:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077527/real 1769077527] req@ffff969e0a46ce00 x1855007923873536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077527.589960:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923873536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077527.589970:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077527.589974:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077527.589978:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077527.589993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077527.590017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077527.590029:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077527/real 1769077527] req@ffff969e0a46ca80 x1855007923873664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077527.590042:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923873664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077527.590051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077527.590055:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077527.590059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077527.590065:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077527.590086:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077527.590098:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077527/real 1769077527] req@ffff969e0a46df80 x1855007923873792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077527.590110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923873792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077527.590120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077527.590123:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077527.590127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077527.590133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077527.590151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077527.590161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077527/real 1769077527] req@ffff969e0a46f100 x1855007923873920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077527.590182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923873920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077527.590192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077527.590196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077527.590200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923873920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077527.908730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077528.612834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077528.932726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077529.636801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077529.956735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077530.660839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077530.980746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077531.684753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077532.004736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077532.708747:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077532.708759:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077532.708765:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077532.708776:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4221 00000100:00080000:2.0:1769077532.708788:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077532.708795:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077532.708815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077532.708842:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923874048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077532.708886:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077532.708888:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077532.708890:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077532.708893:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4221 00000100:00080000:2.0:1769077532.708897:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077532.708900:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077532.708911:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077532.708923:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923874176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077532.708931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923874048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077532.708942:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077532.708946:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077532.708947:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077532.708951:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4221 00000100:00080000:2.0:1769077532.708953:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077532.708957:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077532.708962:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077532.708976:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923874304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077532.708983:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077532.708985:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077532.708987:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077532.708990:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4221 00000100:00080000:2.0:1769077532.709001:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077532.709005:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077532.709009:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077532.709010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077532.709027:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923874432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077532.709037:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077532.709040:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077532.709043:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077532.709050:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4221 00000100:00080000:2.0:1769077532.709053:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077532.709056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077532/real 1769077532] req@ffff969e0a46ed80 x1855007923874048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077532.709061:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077532.709080:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077532.709087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923874048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077532.709098:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077532.709103:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923874560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077532.709106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077532.709116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923874048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077532.709155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077532.709173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077532.709181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077532/real 1769077532] req@ffff969e0a46e300 x1855007923874176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077532.709189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923874176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077532.709211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077532.709214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077532.709217:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077532.709221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077532.709235:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077532.709252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077532/real 1769077532] req@ffff969e0a46f100 x1855007923874304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077532.709260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923874304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077532.709265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077532.709268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077532.709270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077532.709274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077532.709286:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077532.709292:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077532/real 1769077532] req@ffff969e0a46df80 x1855007923874432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077532.709299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923874432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077532.709305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077532.709307:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077532.709314:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077532.709319:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077532.709332:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077532.709339:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077532/real 1769077532] req@ffff969e0a46ca80 x1855007923874560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077532.709346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923874560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077532.709351:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077532.709353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077532.709355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077533.028761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077533.732755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077534.052704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077534.756800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077535.076711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077535.780897:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077536.100731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077536.804747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077537.124738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077537.828700:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077537.828712:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077537.828717:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077537.828727:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4227 00000100:00080000:3.0:1769077537.828739:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077537.828747:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077537.828750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077537.828791:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923874688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077537.828815:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077537.828818:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077537.828820:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077537.828823:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4227 00000100:00080000:3.0:1769077537.828825:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077537.828828:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077537.828836:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077537.828849:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923874816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077537.828857:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077537.828866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923874688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077537.828886:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077537.828888:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077537.828895:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4227 00000100:00080000:3.0:1769077537.828899:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077537.828903:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077537.828907:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077537.828918:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923874944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077537.828926:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077537.828929:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077537.828930:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077537.828933:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4227 00000100:00080000:3.0:1769077537.828936:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077537.828938:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077537.828941:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077537.828948:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077537.828952:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923875072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077537.828960:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077537.828964:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077537.828966:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077537.828968:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4227 00000100:00080000:3.0:1769077537.828973:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077537.828977:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077537.828990:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077537/real 1769077537] req@ffff969e0a7fed80 x1855007923874688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077537.828994:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077537.829017:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923874688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077537.829017:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923875200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077537.829053:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077537.829059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077537.829063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923874688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077537.829086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077537.829102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077537.829119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077537/real 1769077537] req@ffff969e0a7fe680 x1855007923874816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077537.829129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923874816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077537.829135:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077537.829137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077537.829140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077537.829144:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077537.829157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077537.829163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077537/real 1769077537] req@ffff969e0a7fce00 x1855007923874944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077537.829170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923874944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077537.829175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077537.829177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077537.829179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923874944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077537.829183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077537.829193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077537.829199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077537/real 1769077537] req@ffff969e0a7fc700 x1855007923875072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077537.829206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923875072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077537.829217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077537.829219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077537.829222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077537.829226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077537.829238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077537.829244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077537/real 1769077537] req@ffff969e0a7fea00 x1855007923875200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077537.829253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923875200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077537.829262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077537.829266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077537.829269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077538.148753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077538.852747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077539.172791:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077539.876834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077540.196727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077540.900784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077541.220675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077541.924780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077542.244749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077542.948715:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077542.948726:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077542.948732:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077542.948744:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4232 00000100:00080000:3.0:1769077542.948758:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077542.948766:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077542.948807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077542.948820:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923875328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077542.948846:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077542.948849:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077542.948851:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077542.948854:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4232 00000100:00080000:3.0:1769077542.948857:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077542.948860:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077542.948870:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077542.948881:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923875456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077542.948890:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077542.948892:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923875328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077542.948894:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077542.948897:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077542.948904:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4232 00000100:00080000:3.0:1769077542.948926:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077542.948929:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077542.948933:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077542.948951:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923875584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077542.948962:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077542.948965:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077542.948967:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077542.948970:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4232 00000100:00080000:3.0:1769077542.948973:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077542.948976:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077542.948981:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077542.948981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077542.948999:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923875712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077542.949007:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077542.949011:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077542.949013:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077542.949018:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4232 00000100:00080000:3.0:1769077542.949022:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077542.949027:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077542.949032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077542/real 1769077542] req@ffff969e0a7fdf80 x1855007923875328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077542.949035:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077542.949054:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923875840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077542.949062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923875328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077542.949086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077542.949093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077542.949101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923875328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077542.949139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077542.949165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077542.949178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077542/real 1769077542] req@ffff969e0a7fc380 x1855007923875456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077542.949191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923875456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077542.949200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077542.949204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077542.949208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077542.949215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077542.949234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077542.949244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077542/real 1769077542] req@ffff969e0a7fd180 x1855007923875584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077542.949256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923875584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077542.949265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077542.949269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077542.949272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077542.949279:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077542.949297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077542.949306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077542/real 1769077542] req@ffff969e0a7fe300 x1855007923875712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077542.949325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923875712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077542.949334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077542.949337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077542.949341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077542.949347:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077542.949365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077542.949373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077542/real 1769077542] req@ffff969e0a7fca80 x1855007923875840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077542.949385:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923875840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077542.949394:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077542.949398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077542.949401:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923875840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077543.268697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077543.972763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077544.292682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077544.996845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077545.316710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077546.020785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077546.340721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077547.044824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077547.364708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077548.068737:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077548.068748:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077548.068753:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077548.068763:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4237 00000100:00080000:3.0:1769077548.068776:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077548.068787:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077548.068823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077548.068838:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923875968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077548.068867:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077548.068871:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077548.068872:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077548.068876:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4237 00000100:00080000:3.0:1769077548.068879:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077548.068883:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077548.068894:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077548.068906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923876096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077548.068917:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923875968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077548.068919:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077548.068925:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077548.068929:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077548.068933:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4237 00000100:00080000:3.0:1769077548.068935:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077548.068937:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077548.068960:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077548.068971:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923876224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077548.068978:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077548.068981:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077548.068982:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077548.068984:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4237 00000100:00080000:3.0:1769077548.068994:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077548.068997:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077548.069000:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077548.069006:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077548.069021:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923876352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077548.069033:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077548.069037:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077548.069039:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077548.069044:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4237 00000100:00080000:3.0:1769077548.069048:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077548.069051:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077548/real 1769077548] req@ffff969e0a7fd880 x1855007923875968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077548.069056:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077548.069065:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077548.069081:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923876480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077548.069086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923875968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077548.069118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077548.069124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077548.069130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923875968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077548.069157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077548.069183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077548.069196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077548/real 1769077548] req@ffff969e0a7fd500 x1855007923876096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077548.069209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923876096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077548.069218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077548.069222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077548.069225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077548.069232:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077548.069252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077548.069262:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077548/real 1769077548] req@ffff969e0a7fdc00 x1855007923876224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077548.069275:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923876224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077548.069284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077548.069287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077548.069291:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077548.069304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077548.069361:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077548.069375:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077548/real 1769077548] req@ffff969e0a7ff800 x1855007923876352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077548.069388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923876352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077548.069397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077548.069401:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077548.069404:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077548.069410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077548.069431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077548.069441:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077548/real 1769077548] req@ffff969e0a7fc000 x1855007923876480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077548.069454:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923876480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077548.069464:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077548.069467:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077548.069471:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077548.388779:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077549.092773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077549.412709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077550.116780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077550.436759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077551.140822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077551.460725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077552.164761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077552.484719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077553.188668:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077553.188679:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077553.188707:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077553.188717:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4242 00000100:00080000:3.0:1769077553.188729:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077553.188736:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077553.188740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077553.188776:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923876608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077553.188801:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077553.188804:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077553.188806:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077553.188809:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4242 00000100:00080000:3.0:1769077553.188812:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077553.188815:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077553.188825:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077553.188836:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923876736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077553.188846:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077553.188847:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923876608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077553.188852:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077553.188856:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077553.188860:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4242 00000100:00080000:3.0:1769077553.188861:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077553.188864:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077553.188867:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077553.188877:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923876864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077553.188892:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077553.188895:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077553.188897:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077553.188900:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4242 00000100:00080000:3.0:1769077553.188904:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077553.188907:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077553.188911:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077553.188924:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077553.188925:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923876992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077553.188934:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077553.188937:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077553.188938:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077553.188941:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4242 00000100:00080000:3.0:1769077553.188944:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077553.188946:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077553.188949:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077553.188957:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923877120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077553.188965:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077553/real 1769077553] req@ffff969e0a7ffb80 x1855007923876608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077553.188991:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923876608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077553.188999:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077553.189003:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077553.189006:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923876608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077553.189029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077553.189044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077553.189053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077553/real 1769077553] req@ffff969e0a7fc000 x1855007923876736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077553.189061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923876736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077553.189067:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077553.189080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077553.189082:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077553.189085:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077553.189095:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077553.189100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077553/real 1769077553] req@ffff969e0a7ff800 x1855007923876864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077553.189107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923876864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077553.189122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077553.189124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077553.189127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077553.189131:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077553.189156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077553.189163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077553/real 1769077553] req@ffff969e0a7fdc00 x1855007923876992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077553.189170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923876992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077553.189176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077553.189178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077553.189179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923876992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077553.189183:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077553.189193:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077553.189199:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077553/real 1769077553] req@ffff969e0a7fd500 x1855007923877120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077553.189206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923877120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077553.189212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077553.189213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077553.189220:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077553.508681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077554.212746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077554.532698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077555.236775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077555.556699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077556.260774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077556.580745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077557.284782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077557.604731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077558.308736:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077558.308745:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077558.308750:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077558.308759:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4247 00000100:00080000:0.0:1769077558.308772:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077558.308779:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077558.308799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077558.308818:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923877248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077558.308874:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077558.308878:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077558.308880:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077558.308887:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4247 00000100:00080000:0.0:1769077558.308892:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077558.308896:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077558.308905:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077558.308913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923877248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077558.308921:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923877376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077558.308931:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077558.308933:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077558.308935:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077558.308940:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4247 00000100:00080000:0.0:1769077558.308944:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077558.308947:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077558.308952:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077558.308963:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923877504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077558.308968:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077558.308971:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077558.308973:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077558.308977:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4247 00000100:00080000:0.0:1769077558.308986:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077558.308989:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077558.308993:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077558.309001:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923877632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077558.309010:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077558.309020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077558.309021:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077558.309024:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077558.309030:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4247 00000100:00080000:0.0:1769077558.309033:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077558.309037:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077558.309043:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077558.309056:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923877760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077558.309069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077558/real 1769077558] req@ffff969e06a60a80 x1855007923877248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077558.309087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923877248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077558.309104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077558.309109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077558.309126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923877248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077558.309178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077558.309205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077558.309220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077558/real 1769077558] req@ffff969e06a60000 x1855007923877376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077558.309234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923877376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077558.309245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077558.309249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077558.309252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077558.309260:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077558.309283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077558.309296:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077558/real 1769077558] req@ffff969e06a63800 x1855007923877504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077558.309309:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923877504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077558.309318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077558.309322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077558.309326:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077558.309333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077558.309354:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077558.309369:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077558/real 1769077558] req@ffff969e06a60700 x1855007923877632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077558.309378:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923877632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077558.309384:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077558.309387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077558.309390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077558.309396:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077558.309416:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077558.309428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077558/real 1769077558] req@ffff969e06a62d80 x1855007923877760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077558.309440:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923877760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077558.309450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077558.309454:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077558.309456:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923877760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077558.628720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077559.332777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077559.652727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077560.356789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077560.676702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077561.380852:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077561.700730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077562.404832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077562.724753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077563.428772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077563.428776:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077563.428787:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077563.428793:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077563.428803:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4252 00000100:00080000:3.0:1769077563.428837:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077563.428844:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077563.428884:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923877888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077563.428910:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077563.428912:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077563.428914:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077563.428917:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4252 00000100:00080000:3.0:1769077563.428919:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077563.428921:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077563.428931:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077563.428942:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923878016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077563.428953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923877888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077563.428956:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077563.428967:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077563.428969:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077563.428973:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4252 00000100:00080000:3.0:1769077563.428975:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077563.428978:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077563.428983:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077563.428996:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923878144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077563.429003:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077563.429011:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077563.429013:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077563.429016:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4252 00000100:00080000:3.0:1769077563.429028:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077563.429031:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077563.429036:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077563.429055:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923878272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077563.429062:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077563.429082:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077563.429085:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077563.429088:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4252 00000100:00080000:3.0:1769077563.429092:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077563.429096:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077563.429100:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077563.429106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077563.429118:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923878400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077563.429187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077563.429213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077563.429234:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077563.748695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077564.452804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077564.772702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077565.476805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077565.796701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077566.500755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077566.500922:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769077566.500978:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077566.501000:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077566.501068:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077566.501082:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077566.501094:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077566.501106:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077566.501125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077563/real 1769077566] req@ffff969e0a7fca80 x1855007923878016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077566.501157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923878016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077566.501173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077566.501180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077566.501186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077566.501196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077563/real 1769077566] req@ffff969e0a7fe300 x1855007923878144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077566.501210:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923878144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077566.501220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077566.501224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077566.501241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077566.501249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077563/real 1769077566] req@ffff969e0a7fd180 x1855007923878272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077566.501263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923878272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077566.501271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077566.501275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077566.501303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077566.501311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077563/real 1769077566] req@ffff969e0a7fd880 x1855007923877888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077566.501340:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923877888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077566.501348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077566.501352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077566.501355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923877888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077566.501386:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077563/real 1769077566] req@ffff969e0a7fc380 x1855007923878400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077566.501401:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923878400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077566.501410:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077566.501414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077566.501417:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077566.820708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077567.524746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077567.844698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077568.548815:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077568.548828:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077568.548864:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769077568.548872:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077568.548876:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4257 00000100:00080000:0.0:1769077568.548893:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077568.548902:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077568.548940:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923878528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077568.548978:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077568.549006:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077568.549008:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077568.549015:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4257 00000100:00080000:0.0:1769077568.549019:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077568.549025:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077568.549052:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077568.549065:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923878656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077568.549076:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077568.549080:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077568.549082:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077568.549085:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4257 00000100:00080000:0.0:1769077568.549088:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077568.549094:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077568.549096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923878528:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077568.549099:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077568.549129:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923878784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077568.549177:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077568.549182:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077568.549184:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077568.549190:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4257 00000800:00000100:1.0:1769077568.549195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077568.549197:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077568.549206:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077568.549213:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077568.549230:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923878912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077568.549241:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077568/real 1769077568] req@ffff969e06a61180 x1855007923878528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077568.549242:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077568.549258:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077568.549262:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077568.549267:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4257 00000100:00080000:0.0:1769077568.549272:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077568.549279:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077568.549285:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077568.549289:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923878528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077568.549315:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923879040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077568.549319:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077568.549330:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077568.549338:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923878528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077568.549350:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077568.549376:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077568.549391:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077568/real 1769077568] req@ffff969e06a62680 x1855007923878656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077568.549405:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923878656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077568.549414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077568.549418:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077568.549421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077568.549450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077568.549471:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077568.549482:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077568/real 1769077568] req@ffff969e06a61f80 x1855007923878784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077568.549495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923878784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077568.549503:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077568.549507:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077568.549510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077568.549526:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077568.549549:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077568.549561:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077568/real 1769077568] req@ffff969e06a60380 x1855007923878912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077568.549572:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923878912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077568.549581:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077568.549584:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077568.549587:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923878912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077568.549623:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077568.549646:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077568.549657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077568/real 1769077568] req@ffff969e06a61500 x1855007923879040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077568.549669:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923879040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077568.549680:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077568.549684:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077568.549688:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077568.868729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077569.572836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077569.892726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077570.596801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077570.916694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077571.620730:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077571.940702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077572.644789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077572.964686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077573.668696:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077573.668707:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077573.668711:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077573.668721:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4262 00000100:00080000:0.0:1769077573.668733:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077573.668740:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077573.668757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077573.668784:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923879168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077573.668806:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077573.668809:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077573.668810:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077573.668813:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4262 00000100:00080000:0.0:1769077573.668816:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077573.668819:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077573.668828:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077573.668838:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923879296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077573.668843:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923879168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077573.668848:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077573.668851:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077573.668852:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077573.668855:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4262 00000100:00080000:0.0:1769077573.668857:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077573.668859:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077573.668862:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077573.668890:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923879424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077573.668911:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077573.668913:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077573.668915:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077573.668917:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4262 00000100:00080000:0.0:1769077573.668956:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077573.668959:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077573.668964:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077573.668975:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923879552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077573.668976:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077573.668984:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077573.668987:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077573.668989:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077573.668991:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4262 00000100:00080000:0.0:1769077573.668994:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077573.668997:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077573.669000:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077573.669009:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923879680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077573.669016:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077573/real 1769077573] req@ffff969e06a61880 x1855007923879168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077573.669045:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923879168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077573.669054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077573.669058:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077573.669061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923879168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077573.669085:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077573.669105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077573.669125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077573/real 1769077573] req@ffff969e06a63100 x1855007923879296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077573.669139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923879296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077573.669146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077573.669148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077573.669151:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077573.669155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077573.669166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077573.669173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077573/real 1769077573] req@ffff969e06a61500 x1855007923879424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077573.669180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923879424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077573.669194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077573.669197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077573.669199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077573.669203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077573.669217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077573.669224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077573/real 1769077573] req@ffff969e06a60380 x1855007923879552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077573.669231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923879552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077573.669236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077573.669238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077573.669240:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077573.669243:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077573.669253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077573.669258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077573/real 1769077573] req@ffff969e06a61f80 x1855007923879680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077573.669265:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923879680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077573.669271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077573.669272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077573.669279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077573.988746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077574.692747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077575.012761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077575.716737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077576.036740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077576.740782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077577.060733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077577.764734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077578.084771:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077578.788695:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077578.788710:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077578.788737:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1769077578.788749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077578.788749:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4267 00000100:00080000:2.0:1769077578.788762:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077578.788771:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077578.788825:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923879808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077578.788849:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077578.788852:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077578.788853:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077578.788856:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4267 00000100:00080000:2.0:1769077578.788860:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077578.788864:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077578.788875:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077578.788886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923879808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077578.788888:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923879936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077578.788903:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077578.788905:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077578.788907:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077578.788909:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4267 00000100:00080000:2.0:1769077578.788911:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077578.788914:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077578.788917:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077578.788926:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923880064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077578.788945:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077578.788948:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077578.788950:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077578.788952:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4267 00000100:00080000:2.0:1769077578.788956:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077578.788958:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077578.788962:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077578.788970:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923880192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077578.788976:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077578.788978:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077578.788980:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077578.788985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077578.788986:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4267 00000100:00080000:2.0:1769077578.788992:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077578.788994:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077578.788997:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077578.789005:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923880320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077578.789035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077578/real 1769077578] req@ffff969e0a46ce00 x1855007923879808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077578.789053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923879808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077578.789062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077578.789065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077578.789069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923879808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077578.789092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077578.789110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077578.789119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077578/real 1769077578] req@ffff969e0a46f800 x1855007923879936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077578.789127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923879936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077578.789132:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077578.789134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077578.789136:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923879936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077578.789140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077578.789150:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077578.789168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077578/real 1769077578] req@ffff969e0a46d880 x1855007923880064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077578.789176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923880064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077578.789182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077578.789184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077578.789187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077578.789191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077578.789203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077578.789209:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077578/real 1769077578] req@ffff969e0a46c380 x1855007923880192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077578.789217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923880192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077578.789227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077578.789230:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077578.789234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077578.789240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077578.789262:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077578.789274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077578/real 1769077578] req@ffff969e0a46ea00 x1855007923880320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077578.789287:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923880320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077578.789305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077578.789309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077578.789313:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077579.108745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077579.812749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077580.132694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077580.836852:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077581.156745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077581.860844:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077582.180736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077582.884793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077583.204728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077583.908737:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077583.908750:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077583.908755:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077583.908767:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4273 00000100:00080000:3.0:1769077583.908779:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077583.908787:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077583.908792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077583.908828:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923880448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077583.908873:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077583.908877:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077583.908878:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077583.908882:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4273 00000100:00080000:3.0:1769077583.908886:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077583.908891:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077583.908900:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077583.908913:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923880448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077583.908914:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923880576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077583.908929:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077583.908931:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077583.908932:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077583.908935:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4273 00000100:00080000:3.0:1769077583.908937:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077583.908940:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077583.908943:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077583.908951:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923880704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077583.908957:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077583.908959:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077583.908961:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077583.908963:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4273 00000100:00080000:3.0:1769077583.908979:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077583.908983:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077583.908987:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077583.908997:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077583.909000:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923880832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077583.909009:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077583.909011:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077583.909013:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077583.909016:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4273 00000100:00080000:3.0:1769077583.909020:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077583.909023:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077583.909026:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077583.909033:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923880960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077583.909040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077583/real 1769077583] req@ffff969e0a7fdf80 x1855007923880448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077583.909064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923880448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077583.909075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077583.909080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077583.909083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923880448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077583.909104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077583.909123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077583.909132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077583/real 1769077583] req@ffff969e0a7fea00 x1855007923880576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077583.909140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923880576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077583.909146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077583.909148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077583.909151:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077583.909163:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077583.909177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077583.909184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077583/real 1769077583] req@ffff969e0a7fc700 x1855007923880704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077583.909192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923880704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077583.909197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077583.909199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077583.909201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077583.909204:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077583.909215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077583.909220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077583/real 1769077583] req@ffff969e0a7fce00 x1855007923880832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077583.909227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923880832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077583.909232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077583.909234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077583.909236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077583.909239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077583.909248:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077583.909253:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077583/real 1769077583] req@ffff969e0a7fe680 x1855007923880960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077583.909274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923880960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077583.909283:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077583.909287:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077583.909290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923880960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077584.228761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077584.932811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077585.252771:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077585.956787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077586.276724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077586.980778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077587.300773:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077588.004826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077588.324726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077589.028734:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077589.028745:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077589.028750:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077589.028763:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4278 00000100:00080000:3.0:1769077589.028776:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077589.028784:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077589.028805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077589.028837:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923881088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077589.028866:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077589.028890:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077589.028893:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077589.028896:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4278 00000100:00080000:3.0:1769077589.028900:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077589.028908:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923881088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077589.028911:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077589.028927:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077589.028947:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923881216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077589.028960:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077589.028965:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077589.028966:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077589.028971:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4278 00000100:00080000:3.0:1769077589.028979:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077589.028984:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077589.028992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077589.028995:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077589.029015:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923881344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077589.029023:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077589.029026:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077589.029028:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077589.029032:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4278 00000100:00100000:1.0:1769077589.029036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077589/real 1769077589] req@ffff969e0a7fed80 x1855007923881088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077589.029039:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077589.029050:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077589.029064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923881088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077589.029064:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077589.029081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077589.029083:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923881472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077589.029092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077589.029110:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923881088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077589.029112:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077589.029126:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077589.029128:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077589.029135:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4278 00000100:00080000:3.0:1769077589.029139:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077589.029142:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077589.029149:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077589.029153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077589.029166:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923881600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077589.029179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077589.029195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077589/real 1769077589] req@ffff969e0a7fe680 x1855007923881216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077589.029205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923881216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077589.029229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077589.029232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077589.029235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077589.029240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077589.029255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077589.029262:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077589/real 1769077589] req@ffff969e0a7fce00 x1855007923881344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077589.029270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923881344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077589.029275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077589.029277:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077589.029279:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077589.029283:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077589.029294:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077589.029300:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077589/real 1769077589] req@ffff969e0a7fc700 x1855007923881472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077589.029308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923881472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077589.029314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077589.029315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077589.029322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077589.029335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077589.029349:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077589.029356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077589/real 1769077589] req@ffff969e0a7fea00 x1855007923881600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077589.029363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923881600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077589.029369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077589.029371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077589.029373:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077589.348753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077590.052831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077590.372676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077591.077564:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077591.396755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077592.100944:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077592.420735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077593.125634:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077593.444702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077594.149011:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077594.149028:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077594.149034:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077594.149048:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4283 00000100:00080000:2.0:1769077594.149089:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077594.149099:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077594.149140:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923881728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077594.149175:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077594.149179:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077594.149181:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077594.149187:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4283 00000100:00080000:2.0:1769077594.149191:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077594.149196:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077594.149208:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077594.149227:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923881856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077594.149238:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077594.149242:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077594.149244:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077594.149249:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4283 00000100:00080000:2.0:1769077594.149253:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077594.149257:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077594.149264:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077594.149278:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923881984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077594.149290:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077594.149301:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077594.149303:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077594.149307:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4283 00000100:00080000:2.0:1769077594.149311:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077594.149317:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077594.149324:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077594.149338:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923882112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077594.149349:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077594.149353:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077594.149355:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077594.149359:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4283 00000100:00080000:2.0:1769077594.149363:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077594.149367:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077594.149371:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077594.149383:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923882240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769077594.149758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077594.149766:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923881728:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769077594.149861:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077594.149897:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077594/real 1769077594] req@ffff969e0a46d500 x1855007923881728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077594.149918:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923881728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077594.149929:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077594.149932:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077594.149936:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923881728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077594.149945:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077594.149970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077594.149998:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077594/real 1769077594] req@ffff969e0a46f480 x1855007923881856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077594.150012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923881856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077594.150022:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077594.150025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077594.150028:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077594.150035:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077594.150056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077594.150068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077594/real 1769077594] req@ffff969e0a46ea00 x1855007923881984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077594.150079:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923881984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077594.150087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077594.150090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077594.150093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923881984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077594.150099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077594.150116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077594.150125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077594/real 1769077594] req@ffff969e0a46c380 x1855007923882112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077594.150137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923882112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077594.150153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077594.150157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077594.150161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077594.150168:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077594.150189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077594.150197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077594/real 1769077594] req@ffff969e0a46d880 x1855007923882240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077594.150205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923882240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077594.150211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077594.150213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077594.150215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077594.468753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077595.173409:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077595.492752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077596.198136:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077596.516676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077597.220791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077597.540702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077598.244809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077598.564750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077599.268734:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077599.268746:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077599.268751:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077599.268762:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4288 00000100:00080000:2.0:1769077599.268780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077599.268788:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077599.268826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077599.268835:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923882368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077599.268881:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077599.268884:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077599.268886:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077599.268890:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4288 00000100:00080000:2.0:1769077599.268893:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077599.268897:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077599.268907:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077599.268919:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923882496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077599.268928:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077599.268930:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923882368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077599.268931:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077599.268935:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077599.268942:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4288 00000100:00080000:2.0:1769077599.268945:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077599.268947:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077599.268950:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077599.268958:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923882624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077599.268964:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077599.268967:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077599.268968:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077599.268971:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4288 00000100:00080000:2.0:1769077599.268980:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077599.268983:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077599.268988:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077599.268995:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923882752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077599.269003:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077599.269006:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077599.269007:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077599.269009:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4288 00000100:00080000:2.0:1769077599.269013:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077599.269019:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077599.269020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769077599.269029:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077599.269044:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923882880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077599.269071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077599/real 1769077599] req@ffff969e0a46f800 x1855007923882368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077599.269093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923882368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077599.269104:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077599.269109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077599.269113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923882368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077599.269151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077599.269176:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077599.269190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077599/real 1769077599] req@ffff969e0a46ce00 x1855007923882496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077599.269203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923882496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077599.269211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077599.269215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077599.269218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077599.269225:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077599.269245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077599.269256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077599/real 1769077599] req@ffff969e0a46ca80 x1855007923882624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077599.269269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923882624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077599.269277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077599.269281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077599.269285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077599.269291:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077599.269313:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077599.269325:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077599/real 1769077599] req@ffff969e0a46df80 x1855007923882752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077599.269346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923882752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077599.269355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077599.269359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077599.269362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077599.269369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077599.269391:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077599.269402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077599/real 1769077599] req@ffff969e0a46f100 x1855007923882880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077599.269414:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923882880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077599.269423:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077599.269427:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077599.269430:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923882880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077599.588744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077600.292918:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077600.612754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077601.316804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077601.636688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077602.340765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077602.660754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077603.364802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077603.684676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077604.388695:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077604.388710:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077604.388716:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077604.388736:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4293 00000100:00080000:3.0:1769077604.388750:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077604.388758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077604.388760:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077604.388814:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923883008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077604.388848:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077604.388852:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077604.388854:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077604.388860:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4293 00000100:00080000:3.0:1769077604.388884:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077604.388889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923883008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077604.388893:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077604.388909:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077604.388926:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923883136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077604.388938:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077604.388942:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077604.388943:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077604.388947:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4293 00000100:00080000:3.0:1769077604.388952:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077604.388956:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077604.388961:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077604.388969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077604.388975:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923883264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077604.388986:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077604.388990:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077604.388992:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077604.388997:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4293 00000100:00080000:3.0:1769077604.389001:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077604.389006:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077604.389009:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077604/real 1769077604] req@ffff969e0a7fdf80 x1855007923883008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077604.389014:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077604.389033:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923883392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077604.389034:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923883008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077604.389077:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077604.389084:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077604.389086:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077604.389087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077604.389095:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4293 00000100:00080000:1.0:1769077604.389096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077604.389103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923883008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077604.389104:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077604.389113:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077604.389118:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077604.389134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077604.389136:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923883520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077604.389159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077604.389171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077604/real 1769077604] req@ffff969e0a7fc380 x1855007923883136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077604.389180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923883136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077604.389187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077604.389189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077604.389192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077604.389195:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077604.389206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077604.389212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077604/real 1769077604] req@ffff969e0a7fd880 x1855007923883264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077604.389220:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923883264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077604.389225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077604.389227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077604.389229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077604.389239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077604.389253:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077604.389259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077604/real 1769077604] req@ffff969e0a7fd180 x1855007923883392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077604.389267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923883392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077604.389273:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077604.389275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077604.389277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077604.389287:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077604.389297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077604.389303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077604/real 1769077604] req@ffff969e0a7fe300 x1855007923883520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077604.389311:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923883520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077604.389317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077604.389318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077604.389320:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077604.708728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077605.412818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077605.732690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077606.436814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077606.756749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077607.460793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077607.780722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077608.484808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077608.804761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077609.508724:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077609.508739:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077609.508746:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077609.508781:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4298 00000100:00080000:0.0:1769077609.508795:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769077609.508803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077609.508804:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077609.508867:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923883648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077609.508896:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077609.508899:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077609.508902:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077609.508906:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4298 00000100:00080000:0.0:1769077609.508911:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077609.508916:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077609.508928:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077609.508941:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923883648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077609.508943:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923883776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077609.508959:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077609.508963:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077609.508965:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077609.508968:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4298 00000100:00080000:0.0:1769077609.508972:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077609.508975:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077609.508981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077609.508992:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923883904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077609.509015:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077609.509019:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077609.509021:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077609.509025:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4298 00000100:00080000:0.0:1769077609.509030:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077609.509034:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077609.509039:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769077609.509042:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077609.509058:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923884032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077609.509068:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077609.509072:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077609.509075:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077609.509079:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4298 00000100:00100000:1.0:1769077609.509084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077609/real 1769077609] req@ffff969e06a62680 x1855007923883648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077609.509086:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077609.509099:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077609.509107:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077609.509120:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923884160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077609.509125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923883648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077609.509146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077609.509153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077609.509159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923883648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077609.509199:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077609.509221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077609.509231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077609/real 1769077609] req@ffff969e06a61180 x1855007923883776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077609.509240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923883776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077609.509247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077609.509249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077609.509251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077609.509256:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077609.509267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077609.509274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077609/real 1769077609] req@ffff969e06a62d80 x1855007923883904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077609.509282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923883904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077609.509322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077609.509327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077609.509331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923883904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077609.509338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077609.509364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077609.509378:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077609/real 1769077609] req@ffff969e06a60700 x1855007923884032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077609.509391:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923884032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077609.509401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077609.509405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077609.509419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077609.509427:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077609.509449:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077609.509461:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077609/real 1769077609] req@ffff969e06a63800 x1855007923884160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077609.509474:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923884160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077609.509484:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077609.509487:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077609.509497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077609.828751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077610.532733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077610.852726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077611.556740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077611.876690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077612.580770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077612.900729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077613.604825:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077613.924766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077614.628723:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077614.628736:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077614.628741:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077614.628752:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4303 00000100:00080000:3.0:1769077614.628765:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769077614.628771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077614.628773:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077614.628826:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923884288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077614.628859:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077614.628862:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077614.628883:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077614.628887:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4303 00000100:00080000:3.0:1769077614.628891:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077614.628902:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077614.628905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923884288:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077614.628915:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077614.628939:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923884416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077614.628951:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077614.628954:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077614.628956:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077614.628959:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4303 00000100:00080000:3.0:1769077614.628961:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077614.628964:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077614.628967:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077614.628977:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923884544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077614.628985:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077614.628987:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077614.628989:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077614.628991:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4303 00000100:00080000:3.0:1769077614.628994:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077614.628996:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077614.629002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077614.629003:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077614.629024:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923884672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077614.629032:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077614.629036:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077614.629038:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077614.629041:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077614/real 1769077614] req@ffff969e0a7fca80 x1855007923884288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077614.629045:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4303 00000100:00080000:3.0:1769077614.629056:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077614.629066:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077614.629072:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077614.629079:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923884288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077614.629084:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923884800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077614.629114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077614.629122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077614.629127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923884288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077614.629151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077614.629168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077614.629176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077614/real 1769077614] req@ffff969e0a7fd500 x1855007923884416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077614.629184:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923884416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077614.629190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077614.629192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077614.629194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077614.629198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077614.629226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077614.629251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077614/real 1769077614] req@ffff969e0a7fdc00 x1855007923884544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077614.629260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923884544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077614.629267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077614.629271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077614.629274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077614.629281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077614.629303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077614.629315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077614/real 1769077614] req@ffff969e0a7ff800 x1855007923884672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077614.629327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923884672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077614.629335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077614.629338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077614.629342:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077614.629348:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077614.629365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077614.629374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077614/real 1769077614] req@ffff969e0a7fc000 x1855007923884800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077614.629395:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923884800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077614.629415:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077614.629419:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077614.629422:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923884800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077614.948732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077615.652789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077615.972727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077616.676729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077616.996679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077617.700766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077618.020728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077618.724778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077619.044682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077619.748709:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077619.748727:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077619.748734:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077619.748745:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4308 00000100:00080000:3.0:1769077619.748757:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077619.748765:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077619.748781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077619.748810:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923884928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077619.748834:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077619.748854:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077619.748856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077619.748861:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4308 00000100:00080000:3.0:1769077619.748867:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077619.748873:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077619.748885:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077619.748894:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923884928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077619.748906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923885056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077619.748915:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077619.748917:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077619.748919:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077619.748921:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4308 00000100:00080000:3.0:1769077619.748924:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077619.748926:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077619.748930:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077619.748940:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923885184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077619.748947:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077619.748950:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077619.748951:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077619.748953:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4308 00000100:00080000:3.0:1769077619.748955:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077619.748957:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077619.748966:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077619.748975:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923885312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077619.748982:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077619.748985:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077619.748986:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769077619.748991:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077619.748992:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4308 00000100:00080000:3.0:1769077619.748998:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077619.749001:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077619.749004:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077619.749013:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923885440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077619.749039:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077619/real 1769077619] req@ffff969e0a7ffb80 x1855007923884928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077619.749060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923884928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077619.749072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077619.749078:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077619.749082:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923884928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077619.749109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077619.749152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077619.749164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077619/real 1769077619] req@ffff969e0a7fc000 x1855007923885056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077619.749176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923885056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077619.749185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077619.749188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077619.749191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077619.749205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077619.749225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077619.749235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077619/real 1769077619] req@ffff969e0a7ff800 x1855007923885184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077619.749246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923885184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077619.749254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077619.749258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077619.749262:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077619.749269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077619.749289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077619.749299:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077619/real 1769077619] req@ffff969e0a7fdc00 x1855007923885312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077619.749310:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923885312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077619.749318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077619.749322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077619.749325:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077619.749330:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077619.749346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077619.749354:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077619/real 1769077619] req@ffff969e0a7fd500 x1855007923885440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077619.749372:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923885440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077619.749381:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077619.749384:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077619.749387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077620.068733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077620.773015:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077621.092683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077621.796823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077622.116753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077622.820845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077623.140654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077623.844812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077624.164728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077624.868724:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077624.868736:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077624.868741:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077624.868755:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4314 00000100:00080000:0.0:1769077624.868792:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077624.868800:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077624.868808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077624.868847:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923885568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077624.868872:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077624.868875:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077624.868876:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077624.868880:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4314 00000100:00080000:0.0:1769077624.868884:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077624.868887:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077624.868897:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077624.868911:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923885696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077624.868915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923885568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077624.868932:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077624.868936:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077624.868939:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077624.868945:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4314 00000100:00080000:0.0:1769077624.868949:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077624.868954:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077624.868960:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077624.868976:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923885824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077624.868984:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769077624.868989:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077624.868995:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077624.868998:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077624.869002:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4314 00000100:00080000:0.0:1769077624.869006:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077624.869010:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077624.869017:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077624.869026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077624/real 1769077624] req@ffff969e06a60000 x1855007923885568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077624.869049:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923885952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077624.869060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923885568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077624.869067:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077624.869072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077624.869073:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077624.869079:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077624.869081:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769077624.869090:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4314 00000100:00100000:1.0:1769077624.869090:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923885568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077624.869099:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077624.869105:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077624.869110:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077624.869122:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077624.869123:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923886080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077624.869147:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077624.869159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077624/real 1769077624] req@ffff969e06a60a80 x1855007923885696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077624.869167:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923885696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077624.869183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077624.869186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077624.869189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077624.869194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077624.869208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077624.869215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077624/real 1769077624] req@ffff969e06a63800 x1855007923885824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077624.869223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923885824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077624.869229:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077624.869231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077624.869233:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077624.869237:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077624.869247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077624.869252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077624/real 1769077624] req@ffff969e06a60700 x1855007923885952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077624.869259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923885952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077624.869265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077624.869267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077624.869274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923885952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077624.869286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077624.869299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077624.869306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077624/real 1769077624] req@ffff969e06a62d80 x1855007923886080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077624.869314:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923886080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077624.869320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077624.869321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077624.869323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077625.188642:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077625.892768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077626.212753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077626.916804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077627.236677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077627.940770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077628.260724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077628.964818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077629.284658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077629.988731:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077629.988760:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077629.988766:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077629.988778:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4319 00010000:00010000:2.0:1769077629.988809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077629.988825:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077629.988836:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077629.988876:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923886208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077629.988910:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077629.988915:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077629.988918:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077629.988925:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4319 00000100:00080000:3.0:1769077629.988931:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077629.988937:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077629.988944:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923886208:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077629.988966:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077629.988990:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923886336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077629.989003:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077629.989008:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077629.989011:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077629.989016:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4319 00000100:00080000:3.0:1769077629.989021:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077629.989027:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077629.989035:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077629.989053:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923886464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077629.989057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886336:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769077629.989076:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077629.989090:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077629.989094:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769077629.989095:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886464:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769077629.989122:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4319 00000100:00080000:3.0:1769077629.989133:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077629.989139:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077629.989147:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077629.989163:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923886592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077629.989183:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077629.989188:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077629.989190:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077629.989210:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4319 00000100:00080000:3.0:1769077629.989216:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077629.989219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886592:192.168.204.143@tcp:8:kworker.0 00000100:00080000:3.0:1769077629.989224:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077629.989234:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077629.989251:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923886720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077629.989273:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077630.308699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077631.012807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077631.332732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077632.036782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077632.356743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077633.060804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769077633.060965:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769077633.061016:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769077633.061040:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077633.061093:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077633.061103:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077633.061114:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769077633.061122:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077633.061228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077629/real 1769077633] req@ffff969e0a7fc380 x1855007923886720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077633.061260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923886720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077633.061293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077633.061299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077633.061304:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077633.061313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077629/real 1769077633] req@ffff969e0a7fd880 x1855007923886592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077633.061327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923886592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077633.061335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077633.061338:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077633.061341:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077633.061349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077629/real 1769077633] req@ffff969e0a7fd180 x1855007923886464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077633.061362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923886464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077633.061370:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077633.061373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077633.061377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077633.061384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077629/real 1769077633] req@ffff969e0a7fe300 x1855007923886336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077633.061394:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923886336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077633.061410:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077633.061414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077633.061419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077633.061426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077629/real 1769077633] req@ffff969e0a7fca80 x1855007923886208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077633.061440:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923886208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077633.061449:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077633.061453:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077633.061457:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923886208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077633.380704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077634.084769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077634.404725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077635.108776:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077635.108786:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077635.108790:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077635.108800:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4324 00010000:00010000:2.0:1769077635.108803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077635.108812:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077635.108822:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077635.108852:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923886848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077635.108917:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077635.108922:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077635.108933:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077635.108943:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4324 00000100:00080000:0.0:1769077635.108948:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077635.108954:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077635.108965:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077635.108982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923886848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077635.108985:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923886976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077635.109010:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077635.109016:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077635.109019:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077635.109029:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4324 00000100:00080000:0.0:1769077635.109037:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077635.109043:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077635.109051:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077635.109086:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923887104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077635.109097:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077635.109116:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077635.109121:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077635.109124:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077635.109132:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4324 00000100:00080000:0.0:1769077635.109140:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077635.109151:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077635.109155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077635/real 1769077635] req@ffff969e06a61180 x1855007923886848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077635.109161:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077635.109190:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923887232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077635.109200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923886848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077635.109224:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077635.109232:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077635.109233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077635.109243:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077635.109250:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4324 00000100:00080000:0.0:1769077635.109255:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077635.109266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769077635.109268:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077635.109277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923886848:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077635.109280:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077635.109304:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923887360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077635.109323:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077635.109373:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077635.109388:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077635/real 1769077635] req@ffff969e06a62680 x1855007923886976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077635.109402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923886976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077635.109412:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077635.109417:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077635.109420:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923886976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077635.109427:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077635.109447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077635.109465:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077635/real 1769077635] req@ffff969e06a61f80 x1855007923887104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077635.109478:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923887104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077635.109487:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077635.109491:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077635.109495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077635.109502:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077635.109521:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077635.109538:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077635/real 1769077635] req@ffff969e06a60380 x1855007923887232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077635.109565:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923887232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077635.109577:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077635.109580:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077635.109584:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077635.109633:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077635.109668:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077635.109682:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077635/real 1769077635] req@ffff969e06a61500 x1855007923887360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077635.109694:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923887360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077635.109704:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077635.109707:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077635.109711:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077635.428696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077636.132835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077636.452685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077637.156791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077637.476703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077638.180762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077638.500706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077639.204811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077639.524701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077640.228720:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077640.228733:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077640.228739:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077640.228752:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4329 00000100:00080000:3.0:1769077640.228765:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077640.228786:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077640.228814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077640.228837:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923887488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077640.228876:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077640.228879:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077640.228882:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077640.228887:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4329 00000100:00080000:3.0:1769077640.228891:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077640.228896:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077640.228908:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077640.228912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923887488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077640.228924:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923887616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077640.228936:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077640.228940:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077640.228942:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077640.228946:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4329 00000100:00080000:3.0:1769077640.228949:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077640.228953:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077640.228974:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077640.228986:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923887744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077640.228998:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077640.228998:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077640.229004:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077640.229006:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077640.229011:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4329 00000100:00080000:3.0:1769077640.229016:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077640.229021:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077640.229027:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077640.229036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077640/real 1769077640] req@ffff969e0a7fdf80 x1855007923887488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077640.229044:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923887872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077640.229055:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077640.229056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923887488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077640.229063:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077640.229067:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077640.229072:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4329 00000100:00080000:3.0:1769077640.229078:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077640.229084:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077640.229088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769077640.229089:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077640.229099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077640.229105:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923888000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077640.229108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923887488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077640.229148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077640.229170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077640.229180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077640/real 1769077640] req@ffff969e0a7fea00 x1855007923887616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077640.229188:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923887616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077640.229194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077640.229197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077640.229199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077640.229203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077640.229214:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077640.229219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077640/real 1769077640] req@ffff969e0a7fc700 x1855007923887744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077640.229226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923887744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077640.229232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077640.229233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077640.229235:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077640.229245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077640.229259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077640.229266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077640/real 1769077640] req@ffff969e0a7fce00 x1855007923887872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077640.229273:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923887872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077640.229279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077640.229281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077640.229283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923887872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077640.229286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077640.229296:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077640.229301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077640/real 1769077640] req@ffff969e0a7fe680 x1855007923888000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077640.229308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923888000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077640.229313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077640.229315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077640.229317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077640.548684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077641.252812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077641.572710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077642.276854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077642.596709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077643.300769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077643.620730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077644.324760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077644.644733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077645.348744:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769077645.348745:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077645.348756:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077645.348761:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077645.348786:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4334 00000100:00080000:0.0:1769077645.348796:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077645.348801:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077645.348837:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923888128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077645.348865:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077645.348869:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077645.348871:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077645.348876:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4334 00000100:00080000:0.0:1769077645.348881:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077645.348887:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077645.348898:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077645.348920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923888128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077645.348922:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923888256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077645.348944:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077645.348946:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077645.348947:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077645.348951:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4334 00000100:00080000:0.0:1769077645.348954:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077645.348957:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077645.348962:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077645.348973:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923888384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077645.348991:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077645.348995:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077645.348997:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077645.349001:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4334 00000100:00080000:0.0:1769077645.349007:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077645.349013:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077645.349019:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077645.349027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077645.349036:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923888512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077645.349050:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077645.349053:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077645.349054:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077645.349058:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4334 00000100:00080000:0.0:1769077645.349061:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077645.349064:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077645.349068:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077645.349078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077645/real 1769077645] req@ffff969e06a63100 x1855007923888128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077645.349080:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923888640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077645.349116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923888128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077645.349126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077645.349131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077645.349134:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923888128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077645.349160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077645.349188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077645.349198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077645/real 1769077645] req@ffff969e06a61880 x1855007923888256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077645.349207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923888256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077645.349213:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077645.349216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077645.349218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077645.349223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077645.349234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077645.349240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077645/real 1769077645] req@ffff969e06a61500 x1855007923888384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077645.349247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923888384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077645.349262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077645.349265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077645.349267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077645.349272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077645.349287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077645.349293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077645/real 1769077645] req@ffff969e06a60380 x1855007923888512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077645.349301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923888512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077645.349307:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077645.349309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077645.349310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077645.349314:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077645.349324:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077645.349330:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077645/real 1769077645] req@ffff969e06a61f80 x1855007923888640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077645.349338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923888640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077645.349343:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077645.349345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077645.349351:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077645.668766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077646.372819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077646.692692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077647.396760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077647.716757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077648.420813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077648.740689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077649.444809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077649.764718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077650.468795:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077650.468809:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077650.468815:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077650.468828:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4339 00010000:00010000:0.0:1769077650.468842:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077650.468865:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077650.468873:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077650.468916:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923888768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077650.468949:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077650.468954:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077650.468956:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077650.468961:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4339 00000100:00080000:2.0:1769077650.468966:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077650.468972:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077650.468984:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077650.468997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923888768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077650.469004:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923888896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077650.469020:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077650.469023:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077650.469027:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077650.469031:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4339 00000100:00080000:2.0:1769077650.469036:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077650.469040:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077650.469047:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077650.469064:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923889024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077650.469075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077650.469087:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077650.469100:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077650.469103:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077650.469111:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4339 00000100:00100000:1.0:1769077650.469115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077650/real 1769077650] req@ffff969e0a46e300 x1855007923888768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077650.469120:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077650.469134:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077650.469141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923888768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077650.469143:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077650.469163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077650.469168:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077650.469174:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923889152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077650.469175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923888768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077650.469198:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077650.469203:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077650.469206:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077650.469211:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4339 00000100:00080000:2.0:1769077650.469217:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077650.469220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888896:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077650.469225:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077650.469236:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077650.469247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077650.469259:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923889280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077650.469263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077650/real 1769077650] req@ffff969e0a46ed80 x1855007923888896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077650.469317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923888896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077650.469328:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077650.469336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077650.469340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923888896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077650.469346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077650.469364:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077650.469386:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077650/real 1769077650] req@ffff969e0a46f100 x1855007923889024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077650.469395:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923889024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077650.469401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077650.469404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077650.469406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077650.469410:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077650.469424:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077650.469430:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077650/real 1769077650] req@ffff969e0a46df80 x1855007923889152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077650.469438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923889152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077650.469443:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077650.469445:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077650.469447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077650.469462:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077650.469472:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077650.469478:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077650/real 1769077650] req@ffff969e0a46ca80 x1855007923889280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077650.469487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923889280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077650.469502:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077650.469506:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077650.469510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077650.788703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077651.492740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077651.812712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077652.516849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077652.836759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077653.540720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077653.860687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077654.564766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077654.884748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077655.588714:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077655.588728:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077655.588733:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077655.588747:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4344 00000100:00080000:3.0:1769077655.588760:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077655.588768:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077655.588777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077655.588816:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923889408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077655.588854:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077655.588859:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077655.588861:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077655.588866:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4344 00000100:00080000:3.0:1769077655.588870:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077655.588875:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077655.588886:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077655.588900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923889408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077655.588927:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923889536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077655.588940:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077655.588944:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077655.588947:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077655.588951:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4344 00000100:00080000:3.0:1769077655.588969:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077655.588974:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769077655.588981:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769077655.588982:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077655.588998:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923889664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077655.589009:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077655.589013:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077655.589016:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077655.589021:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4344 00000100:00100000:2.0:1769077655.589026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077655/real 1769077655] req@ffff969e0a7fed80 x1855007923889408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077655.589026:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077655.589035:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077655.589045:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077655.589052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923889408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077655.589059:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923889792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077655.589085:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077655.589090:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077655.589090:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077655.589095:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077655.589098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077655.589103:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923889408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077655.589112:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4344 00000100:00080000:3.0:1769077655.589118:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077655.589123:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077655.589128:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077655.589133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077655.589143:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923889920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077655.589157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077655.589170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077655/real 1769077655] req@ffff969e0a7fe680 x1855007923889536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077655.589180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923889536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077655.589185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077655.589188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077655.589190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077655.589193:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077655.589204:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077655.589210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077655/real 1769077655] req@ffff969e0a7fce00 x1855007923889664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077655.589217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923889664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077655.589223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077655.589224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077655.589226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077655.589236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077655.589250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077655.589256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077655/real 1769077655] req@ffff969e0a7fc700 x1855007923889792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077655.589264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923889792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077655.589270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077655.589272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077655.589274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077655.589285:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077655.589295:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077655.589301:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077655/real 1769077655] req@ffff969e0a7fea00 x1855007923889920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077655.589308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923889920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077655.589314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077655.589315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077655.589317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923889920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077655.908775:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077656.612766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077656.932746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077657.637744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077657.956707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077658.660903:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077658.980718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077659.685051:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077660.004696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077660.708882:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077660.708899:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077660.708906:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077660.708920:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4349 00000100:00080000:3.0:1769077660.708937:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077660.708947:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077660.709003:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923890048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077660.709053:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077660.709058:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077660.709060:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077660.709066:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4349 00000100:00080000:3.0:1769077660.709071:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077660.709076:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077660.709116:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077660.709134:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923890176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077660.709147:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077660.709151:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077660.709155:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077660.709158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923890048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077660.709162:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4349 00000100:00080000:3.0:1769077660.709171:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077660.709178:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077660.709184:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077660.709218:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923890304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077660.709231:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077660.709235:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077660.709238:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077660.709243:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4349 00000100:00080000:3.0:1769077660.709272:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077660.709277:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077660.709283:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077660.709291:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077660.709300:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923890432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077660.709311:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077660.709315:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077660.709326:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077660.709331:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4349 00000100:00080000:3.0:1769077660.709339:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077660.709344:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077660.709351:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077660.709366:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923890560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077660.709371:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077660/real 1769077660] req@ffff969e0a7fdf80 x1855007923890048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077660.709407:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923890048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077660.709421:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077660.709426:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077660.709461:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923890048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077660.709491:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077660.709517:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077660.709531:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077660/real 1769077660] req@ffff969e0a7fca80 x1855007923890176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077660.709546:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923890176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077660.709556:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077660.709559:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077660.709563:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077660.709570:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077660.709623:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077660.709637:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077660/real 1769077660] req@ffff969e0a7fe300 x1855007923890304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077660.709650:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923890304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077660.709671:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077660.709675:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077660.709679:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077660.709686:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077660.709710:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077660.709721:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077660/real 1769077660] req@ffff969e0a7fd180 x1855007923890432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077660.709733:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923890432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077660.709742:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077660.709745:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077660.709749:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077660.709755:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077660.709774:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077660.709784:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077660/real 1769077660] req@ffff969e0a7fd880 x1855007923890560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077660.709796:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923890560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077660.709805:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077660.709808:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077660.709818:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890560:192.168.204.143@tcp:8:kworker.0 00010000:00010000:0.0:1769077660.709853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077661.028731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077661.733069:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077662.052667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077662.757189:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077663.076727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077663.780785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077664.100666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077664.804740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077665.124714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077665.828719:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077665.828730:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077665.828743:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077665.828757:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4355 00000100:00080000:3.0:1769077665.828768:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077665.828776:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077665.828791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077665.828821:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923890688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077665.828845:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077665.828847:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077665.828849:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077665.828852:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4355 00000100:00080000:3.0:1769077665.828854:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077665.828875:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077665.828885:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077665.828892:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923890688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077665.828898:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923890816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077665.828907:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077665.828910:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077665.828911:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077665.828914:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4355 00000100:00080000:3.0:1769077665.828918:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077665.828920:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077665.828923:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077665.828936:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923890944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077665.828946:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077665.828949:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077665.828951:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077665.828956:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4355 00000100:00080000:3.0:1769077665.828961:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077665.828967:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077665.828968:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077665.828977:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077665.828992:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923891072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077665.829001:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077665.829006:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077665/real 1769077665] req@ffff969e0a7fc380 x1855007923890688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077665.829016:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077665.829021:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077665.829027:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4355 00000100:00100000:2.0:1769077665.829032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923890688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077665.829034:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077665.829043:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769077665.829050:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769077665.829051:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769077665.829059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077665.829068:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923891200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077665.829069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923890688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077665.829106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077665.829127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077665.829136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077665/real 1769077665] req@ffff969e0a7fd500 x1855007923890816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077665.829145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923890816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077665.829152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077665.829154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077665.829157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077665.829160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077665.829179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077665.829201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077665/real 1769077665] req@ffff969e0a7fdc00 x1855007923890944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077665.829216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923890944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077665.829225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077665.829228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077665.829232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923890944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077665.829239:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077665.829262:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077665.829274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077665/real 1769077665] req@ffff969e0a7ff800 x1855007923891072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077665.829288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923891072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077665.829296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077665.829298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077665.829301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077665.829305:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077665.829320:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077665.829327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077665/real 1769077665] req@ffff969e0a7fc000 x1855007923891200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077665.829335:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923891200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077665.829373:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077665.829376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077665.829379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077666.148680:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077666.852733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077667.172725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077667.876833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077668.196686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077668.900748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077669.220738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077669.924869:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077670.244711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077670.948688:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077670.948702:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077670.948708:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077670.948720:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4360 00000100:00080000:3.0:1769077670.948735:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077670.948744:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077670.948770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077670.948793:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923891328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077670.948818:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077670.948821:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077670.948823:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077670.948826:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4360 00000100:00080000:3.0:1769077670.948855:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077670.948860:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077670.948866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923891328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077670.948873:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077670.948896:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923891456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077670.948910:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077670.948914:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077670.948916:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077670.948919:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4360 00000100:00080000:3.0:1769077670.948924:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077670.948927:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077670.948931:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769077670.948934:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077670.948945:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923891584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077670.948953:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077670.948956:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077670.948957:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077670.948960:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4360 00000100:00080000:3.0:1769077670.948964:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077670.948967:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077670.948972:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077670/real 1769077670] req@ffff969e0a7ffb80 x1855007923891328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077670.948973:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077670.948987:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923891712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077670.948994:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923891328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077670.949009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077670.949012:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077670.949015:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077670.949019:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077670.949023:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769077670.949026:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4360 00000100:00100000:2.0:1769077670.949031:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923891328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077670.949031:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077670.949040:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077670.949046:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077670.949060:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077670.949063:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923891840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077670.949083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077670.949095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077670/real 1769077670] req@ffff969e0a7fc000 x1855007923891456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077670.949104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923891456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077670.949110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077670.949112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077670.949114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077670.949126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077670.949140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077670.949146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077670/real 1769077670] req@ffff969e0a7ff800 x1855007923891584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077670.949154:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923891584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077670.949160:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077670.949162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077670.949165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077670.949168:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077670.949179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077670.949184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077670/real 1769077670] req@ffff969e0a7fdc00 x1855007923891712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077670.949192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923891712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077670.949197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077670.949199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077670.949201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077670.949211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077670.949221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077670.949227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077670/real 1769077670] req@ffff969e0a7fd500 x1855007923891840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077670.949240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923891840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077670.949246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077670.949248:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077670.949250:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923891840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077671.268711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077671.972817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077672.292703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077672.996732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077673.316731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077674.020854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077674.340703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077675.044826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077675.364767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077676.068726:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077676.068737:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077676.068742:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077676.068753:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4365 00000100:00080000:0.0:1769077676.068772:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077676.068781:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077676.068806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077676.068834:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923891968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077676.068861:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077676.068883:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077676.068886:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077676.068892:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4365 00000100:00080000:0.0:1769077676.068897:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077676.068905:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077676.068907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923891968:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077676.068918:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077676.068938:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923892096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077676.068948:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077676.068951:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077676.068953:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077676.068956:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4365 00000100:00080000:0.0:1769077676.068961:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077676.068965:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077676.068970:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077676.068982:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923892224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077676.068994:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077676.068995:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077676.069001:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077676.069004:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077676.069010:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4365 00000100:00080000:0.0:1769077676.069016:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077676.069021:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077676.069033:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077676.069043:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077676/real 1769077676] req@ffff969e06a62680 x1855007923891968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077676.069047:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923892352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077676.069081:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077676.069085:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077676.069088:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077676.069093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923891968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077676.069096:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4365 00000100:00080000:0.0:1769077676.069104:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077676.069112:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769077676.069113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077676.069118:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769077676.069123:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077676.069129:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923891968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077676.069137:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923892480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077676.069162:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077676.069228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077676.069246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077676/real 1769077676] req@ffff969e06a61180 x1855007923892096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077676.069259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923892096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077676.069284:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077676.069288:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077676.069292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077676.069300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077676.069324:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077676.069337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077676/real 1769077676] req@ffff969e06a62d80 x1855007923892224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077676.069350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923892224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077676.069359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077676.069363:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077676.069367:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077676.069374:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077676.069396:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077676.069408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077676/real 1769077676] req@ffff969e06a60700 x1855007923892352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077676.069422:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923892352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077676.069431:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077676.069435:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077676.069447:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077676.069454:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077676.069477:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077676.069500:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077676/real 1769077676] req@ffff969e06a63800 x1855007923892480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077676.069514:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923892480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077676.069524:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077676.069528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077676.069531:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077676.388721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077677.092833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077677.412689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077678.116839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077678.436743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077679.140829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077679.460702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077680.164800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077680.484718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077681.188753:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077681.188765:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077681.188770:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077681.188782:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4370 00000100:00080000:3.0:1769077681.188796:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077681.188805:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:1.0:1769077681.188811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077681.188864:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923892608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077681.188889:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077681.188892:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077681.188893:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077681.188897:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4370 00000100:00080000:3.0:1769077681.188900:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077681.188904:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077681.188913:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077681.188925:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923892608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077681.188925:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923892736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077681.188939:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077681.188941:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077681.188942:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077681.188944:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4370 00000100:00080000:3.0:1769077681.188948:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077681.188950:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077681.188953:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077681.188961:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923892864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077681.188967:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077681.188970:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077681.188971:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077681.188978:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4370 00000100:00080000:3.0:1769077681.188997:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077681.189000:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077681.189004:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077681.189008:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077681.189019:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923892992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077681.189027:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077681.189030:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077681.189031:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077681.189034:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4370 00000100:00080000:3.0:1769077681.189038:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077681.189042:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077681.189044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077681/real 1769077681] req@ffff969e0a7fc380 x1855007923892608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769077681.189047:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077681.189058:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923893120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077681.189076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923892608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077681.189103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077681.189108:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077681.189113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923892608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077681.189148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077681.189173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077681.189186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077681/real 1769077681] req@ffff969e0a7fd880 x1855007923892736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077681.189198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923892736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077681.189207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077681.189210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077681.189214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077681.189220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077681.189236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077681.189245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077681/real 1769077681] req@ffff969e0a7fd180 x1855007923892864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077681.189256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923892864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077681.189264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077681.189267:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077681.189270:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077681.189276:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077681.189292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077681.189302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077681/real 1769077681] req@ffff969e0a7fe300 x1855007923892992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077681.189319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923892992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077681.189325:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077681.189327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077681.189330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923892992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077681.189334:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077681.189348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077681.189355:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077681/real 1769077681] req@ffff969e0a7fca80 x1855007923893120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077681.189362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923893120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077681.189368:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077681.189370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077681.189372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077681.508691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077682.212723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077682.532755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077683.236765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077683.556681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077684.260714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077684.580690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077685.284748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077685.604714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077686.308711:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077686.308724:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077686.308729:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769077686.308737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077686.308741:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4375 00000100:00080000:0.0:1769077686.308755:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077686.308765:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077686.308807:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923893248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077686.308832:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077686.308835:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077686.308837:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077686.308840:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4375 00000100:00080000:0.0:1769077686.308862:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077686.308866:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077686.308875:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077686.308886:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923893376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077686.308894:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077686.308897:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077686.308899:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077686.308901:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4375 00000100:00080000:0.0:1769077686.308904:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077686.308909:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077686.308909:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923893248:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077686.308917:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077686.308927:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923893504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077686.308935:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077686.308938:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077686.308939:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077686.308942:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4375 00000100:00080000:0.0:1769077686.308945:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077686.308948:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077686.308951:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077686.308958:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923893632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077686.308973:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077686.308976:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077686.308977:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077686.308981:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4375 00000800:00000100:2.0:1769077686.308985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077686.308986:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077686.308992:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077686.308996:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077686.309022:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923893760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077686.309036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077686/real 1769077686] req@ffff969e06a60a80 x1855007923893248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077686.309052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923893248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077686.309071:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077686.309075:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077686.309078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923893248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077686.309084:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077686.309101:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077686.309110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077686/real 1769077686] req@ffff969e06a60000 x1855007923893376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077686.309118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923893376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077686.309123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077686.309125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077686.309127:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077686.309157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077686.309169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077686.309175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077686/real 1769077686] req@ffff969e06a63800 x1855007923893504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077686.309182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923893504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077686.309188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077686.309190:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077686.309192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077686.309201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077686.309215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077686.309222:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077686/real 1769077686] req@ffff969e06a60700 x1855007923893632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077686.309236:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923893632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077686.309242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077686.309245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077686.309247:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077686.309250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077686.309261:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077686.309267:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077686/real 1769077686] req@ffff969e06a62d80 x1855007923893760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077686.309274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923893760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077686.309279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077686.309281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077686.309283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923893760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077686.628725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077687.332748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077687.652673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077688.356672:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077688.676674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077689.380719:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077689.700669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077690.404661:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077690.724695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077691.428740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077691.428748:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077691.428759:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077691.428764:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077691.428796:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4380 00000100:00080000:2.0:1769077691.428808:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077691.428814:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077691.428852:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923893888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077691.428883:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077691.428886:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077691.428887:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077691.428891:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4380 00000100:00080000:2.0:1769077691.428895:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077691.428898:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077691.428907:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077691.428941:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923894016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077691.428950:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923893888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077691.428958:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077691.428962:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077691.428964:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077691.428967:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4380 00000100:00080000:2.0:1769077691.428973:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077691.428979:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077691.428986:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077691.429003:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923894144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077691.429029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077691.429040:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077691.429043:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077691.429046:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077691.429054:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4380 00000100:00080000:2.0:1769077691.429062:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077691.429068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077691/real 1769077691] req@ffff969e0a46ce00 x1855007923893888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077691.429075:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077691.429095:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077691.429099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923893888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077691.429114:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077691.429120:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077691.429121:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923894272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077691.429131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923893888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077691.429152:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077691.429157:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077691.429161:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077691.429166:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4380 00000100:00080000:2.0:1769077691.429172:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077691.429173:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894016:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077691.429180:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077691.429187:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077691.429211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077691.429222:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923894400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077691.429228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077691/real 1769077691] req@ffff969e0a46f800 x1855007923894016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077691.429250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923894016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077691.429259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077691.429263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077691.429266:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077691.429272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077691.429287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077691.429293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077691/real 1769077691] req@ffff969e0a46d880 x1855007923894144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077691.429301:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923894144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077691.429306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077691.429308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077691.429310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077691.429314:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077691.429324:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077691.429330:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077691/real 1769077691] req@ffff969e0a46c380 x1855007923894272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077691.429350:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923894272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077691.429355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077691.429358:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077691.429360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077691.429372:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077691.429384:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077691.429390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077691/real 1769077691] req@ffff969e0a46ea00 x1855007923894400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077691.429397:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923894400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077691.429402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077691.429404:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077691.429406:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077691.748717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077692.452777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077692.772728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077693.476753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077693.796736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077694.500722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077694.820701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077695.524750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077695.844698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077696.548703:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077696.548719:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077696.548725:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769077696.548731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077696.548740:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4385 00000100:00080000:2.0:1769077696.548776:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077696.548783:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077696.548824:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923894528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077696.548864:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077696.548867:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077696.548868:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077696.548872:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4385 00000100:00080000:2.0:1769077696.548877:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077696.548882:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077696.548892:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077696.548906:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923894656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077696.548912:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923894528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077696.548917:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077696.548928:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077696.548931:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077696.548936:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4385 00000100:00080000:2.0:1769077696.548939:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077696.548941:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077696.548944:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077696.548961:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923894784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077696.548971:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077696.548982:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077696.548985:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077696.548990:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4385 00000100:00080000:2.0:1769077696.548996:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077696.549003:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077696.549011:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077696.549045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077696.549055:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923894912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077696.549067:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077696.549084:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077696.549086:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077696.549093:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4385 00000100:00080000:2.0:1769077696.549100:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077696.549105:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077696.549112:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077696.549128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077696.549132:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923895040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077696.549176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077696.549197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077696.868727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077697.572706:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077697.892719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077698.596821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077698.916718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077699.620693:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1769077699.620857:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:3.0:1769077699.620901:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769077699.620920:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077699.620958:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077699.620969:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077699.620981:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077699.620992:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769077699.621003:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077696/real 1769077699] req@ffff969e0a46c380 x1855007923894912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077699.621036:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923894912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077699.621049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077699.621055:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077699.621060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077699.621068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077696/real 1769077699] req@ffff969e0a46d880 x1855007923895040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077699.621082:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923895040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077699.621105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077699.621109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077699.621113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077699.621120:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077696/real 1769077699] req@ffff969e0a46d500 x1855007923894656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077699.621155:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923894656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077699.621183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077699.621187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077699.621191:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077699.621198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077696/real 1769077699] req@ffff969e0a46ea00 x1855007923894784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077699.621211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923894784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077699.621219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077699.621223:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077699.621227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923894784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077699.621234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077696/real 1769077699] req@ffff969e0a46f480 x1855007923894528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077699.621248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923894528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077699.621256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077699.621259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077699.621263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923894528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077699.940706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077700.644728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077700.964650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077701.668728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077701.668844:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077701.668852:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077701.668856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077701.668867:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4390 00000100:00080000:2.0:1769077701.668878:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077701.668886:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077701.668921:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923895168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077701.668964:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077701.668968:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077701.668970:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077701.668974:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4390 00000100:00080000:2.0:1769077701.668978:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077701.668983:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077701.669005:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077701.669018:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923895296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077701.669030:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077701.669033:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077701.669035:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077701.669039:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4390 00000100:00080000:2.0:1769077701.669043:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077701.669046:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077701.669051:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077701.669065:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923895424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077701.669075:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077701.669079:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077701.669081:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077701.669084:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4390 00000100:00080000:2.0:1769077701.669092:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077701.669096:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077701.669101:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077701.669111:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923895552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077701.669121:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077701.669124:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077701.669126:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077701.669130:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4390 00000100:00080000:2.0:1769077701.669132:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077701.669136:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077701.669140:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077701.669148:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923895680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077701.669623:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923895168:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769077701.669704:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077701.669740:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077701/real 1769077701] req@ffff969e0a46f800 x1855007923895168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077701.669779:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923895168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077701.669791:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077701.669795:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077701.669800:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923895168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077701.669808:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077701.669832:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077701.669857:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077701/real 1769077701] req@ffff969e0a46ce00 x1855007923895296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077701.669870:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923895296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077701.669879:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077701.669882:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077701.669885:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077701.669893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077701.669914:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077701.669926:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077701/real 1769077701] req@ffff969e0a46ca80 x1855007923895424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077701.669950:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923895424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077701.669960:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077701.669964:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077701.669967:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077701.669974:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077701.669999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077701.670013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077701/real 1769077701] req@ffff969e0a46df80 x1855007923895552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077701.670028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923895552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077701.670037:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077701.670040:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077701.670044:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077701.670051:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077701.670072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077701.670084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077701/real 1769077701] req@ffff969e0a46f100 x1855007923895680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077701.670097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923895680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077701.670118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077701.670121:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077701.670125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077701.988658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077702.692717:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077703.012649:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077703.716729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077704.036671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077704.740714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077705.060650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077705.764748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077706.084725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077706.788746:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1769077706.788756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077706.788759:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077706.788766:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077706.788777:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4395 00000100:00080000:0.0:1769077706.788789:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077706.788796:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077706.788831:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923895808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077706.788860:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077706.788863:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077706.788865:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077706.788870:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4395 00000100:00080000:0.0:1769077706.788875:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077706.788881:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077706.788908:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923895808:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077706.788915:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077706.788936:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923895936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077706.788949:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077706.788953:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077706.788955:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077706.788959:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4395 00000100:00080000:0.0:1769077706.788966:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077706.788971:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077706.788976:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769077706.788986:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077706.788989:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923896064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077706.789000:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077706.789005:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077706.789008:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077706.789012:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4395 00000100:00080000:0.0:1769077706.789017:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077706.789021:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077706.789026:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077706.789031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077706/real 1769077706] req@ffff969e06a61180 x1855007923895808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077706.789040:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923896192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077706.789058:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077706.789062:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1769077706.789065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923895808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077706.789071:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077706.789080:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4395 00000100:00080000:0.0:1769077706.789087:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077706.789088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077706.789095:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077706.789098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:0.0:1769077706.789103:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077706.789107:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923895808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077706.789118:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923896320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077706.789138:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077706.789178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077706.789192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077706/real 1769077706] req@ffff969e06a62680 x1855007923895936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077706.789207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923895936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077706.789225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077706.789237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077706.789242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923895936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077706.789253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077706.789276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077706.789288:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077706/real 1769077706] req@ffff969e06a61f80 x1855007923896064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077706.789300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923896064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077706.789308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077706.789312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077706.789315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077706.789321:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077706.789343:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077706.789353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077706/real 1769077706] req@ffff969e06a60380 x1855007923896192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077706.789365:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923896192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077706.789374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077706.789377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077706.789388:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077706.789395:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077706.789417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077706.789428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077706/real 1769077706] req@ffff969e06a61500 x1855007923896320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077706.789441:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923896320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077706.789451:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077706.789455:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077706.789458:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077707.108744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077707.812773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077708.132775:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077708.836753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077709.156715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077709.860698:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077710.180716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077710.884725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077711.204700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077711.908678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077711.908688:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1769077711.908692:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077711.908693:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077711.908704:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4401 00000100:00080000:3.0:1769077711.908733:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077711.908740:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077711.908768:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923896448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077711.908789:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077711.908791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077711.908793:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077711.908796:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4401 00000100:00080000:3.0:1769077711.908799:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077711.908802:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077711.908809:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077711.908820:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923896448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077711.908825:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923896576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077711.908833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077711.908835:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077711.908836:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077711.908839:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4401 00000100:00080000:3.0:1769077711.908842:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077711.908844:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077711.908847:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077711.908855:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923896704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077711.908861:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077711.908868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077711.908870:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077711.908873:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4401 00000100:00080000:3.0:1769077711.908876:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077711.908878:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077711.908881:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077711.908883:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077711.908892:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923896832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077711.908899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077711.908902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077711.908903:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077711.908905:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4401 00000100:00080000:3.0:1769077711.908908:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077711.908911:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077711.908914:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077711.908919:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077711/real 1769077711] req@ffff969e0a7fdf80 x1855007923896448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077711.908922:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923896960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077711.908956:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923896448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077711.908975:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077711.908979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077711.908989:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923896448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077711.909010:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077711.909029:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077711.909040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077711/real 1769077711] req@ffff969e0a7fea00 x1855007923896576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077711.909052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923896576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077711.909060:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077711.909063:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077711.909066:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077711.909072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077711.909085:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077711.909093:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077711/real 1769077711] req@ffff969e0a7fc700 x1855007923896704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077711.909104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923896704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077711.909111:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077711.909114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077711.909118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077711.909123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077711.909137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077711.909154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077711/real 1769077711] req@ffff969e0a7fce00 x1855007923896832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077711.909172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923896832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077711.909182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077711.909186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077711.909190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077711.909195:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077711.909211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077711.909219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077711/real 1769077711] req@ffff969e0a7fe680 x1855007923896960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077711.909230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923896960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077711.909238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077711.909241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077711.909244:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923896960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077712.228705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077712.932727:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077713.252719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077713.956712:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077714.276704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077714.980724:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077715.300701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077716.004715:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077716.324697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077717.028688:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077717.028698:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077717.028702:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077717.028729:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4406 00000100:00080000:1.0:1769077717.028740:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077717.028747:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077717.028784:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923897088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077717.028807:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077717.028809:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077717.028810:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077717.028813:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4406 00000100:00080000:1.0:1769077717.028816:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077717.028819:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077717.028827:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077717.028835:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923897216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077717.028843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077717.028845:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077717.028848:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077717.028849:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923897088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077717.028852:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4406 00000100:00080000:1.0:1769077717.028856:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077717.028860:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077717.028863:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077717.028869:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923897344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077717.028884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077717.028886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077717.028887:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077717.028891:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4406 00000100:00080000:1.0:1769077717.028894:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077717.028896:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077717.028899:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077717.028906:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923897472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077717.028911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077717.028914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077717.028915:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077717.028917:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4406 00000100:00080000:1.0:1769077717.028918:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077717.028920:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077717.028923:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077717.028925:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077717.028932:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923897600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077717.028958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077717/real 1769077717] req@ffff969e10b61500 x1855007923897088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1769077717.028973:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077717.028974:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923897088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077717.028989:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077717.028994:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077717.028997:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923897088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077717.029024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077717.029046:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077717.029058:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077717/real 1769077717] req@ffff969e10b61f80 x1855007923897216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077717.029070:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923897216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077717.029075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077717.029077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077717.029079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077717.029083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077717.029096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077717.029102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077717/real 1769077717] req@ffff969e10b60a80 x1855007923897344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077717.029108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923897344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077717.029113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077717.029115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077717.029116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077717.029119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077717.029128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077717.029133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077717/real 1769077717] req@ffff969e10b62a00 x1855007923897472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077717.029145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923897472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077717.029150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077717.029152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077717.029154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077717.029157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077717.029168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077717.029173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077717/real 1769077717] req@ffff969e10b62680 x1855007923897600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077717.029179:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923897600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077717.029184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077717.029185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077717.029187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077717.348723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077718.052758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077718.372713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077719.076719:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077719.396714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077720.100741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077720.420709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077721.124747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077721.444746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077722.148709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077722.148720:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077722.148724:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769077722.148730:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077722.148735:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4411 00000100:00080000:3.0:1769077722.148747:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077722.148753:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077722.148786:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923897728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077722.148810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077722.148814:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077722.148816:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077722.148820:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4411 00000100:00080000:3.0:1769077722.148823:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077722.148827:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077722.148836:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077722.148850:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923897856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077722.148854:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923897728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077722.148868:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077722.148871:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077722.148872:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077722.148877:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4411 00000100:00080000:3.0:1769077722.148881:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077722.148884:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077722.148904:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077722.148917:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923897984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077722.148920:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077722.148930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077722.148934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077722.148936:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077722.148941:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4411 00000100:00080000:3.0:1769077722.148945:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077722.148949:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077722.148954:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077722.148956:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077722/real 1769077722] req@ffff969e0a7fed80 x1855007923897728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077722.148966:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923898112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077722.148976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077722.148979:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923897728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077722.148980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077722.148983:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077722.148991:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4411 00000100:00080000:3.0:1769077722.148995:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077722.148998:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077722.149004:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769077722.149014:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1769077722.149019:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923898240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077722.149022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077722.149030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923897728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077722.149055:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077722.149075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077722.149085:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077722/real 1769077722] req@ffff969e0a7fe680 x1855007923897856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077722.149096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923897856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077722.149103:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077722.149106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077722.149109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077722.149115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077722.149129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077722.149137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077722/real 1769077722] req@ffff969e0a7fce00 x1855007923897984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077722.149147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923897984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077722.149154:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077722.149157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077722.149160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923897984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077722.149171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077722.149186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077722.149194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077722/real 1769077722] req@ffff969e0a7fc700 x1855007923898112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077722.149204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923898112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077722.149212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077722.149215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077722.149218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077722.149223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077722.149236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077722.149244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077722/real 1769077722] req@ffff969e0a7fea00 x1855007923898240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077722.149254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923898240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077722.149262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077722.149272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077722.149276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077722.468673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077723.172694:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077723.492722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077724.197151:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077724.516697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077725.221637:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077725.540711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077726.245266:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077726.564703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077727.269502:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077727.269525:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077727.269531:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077727.269546:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4416 00000100:00080000:1.0:1769077727.269568:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077727.269578:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077727.269687:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923898368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077727.269722:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077727.269726:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077727.269728:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077727.269733:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4416 00000100:00080000:1.0:1769077727.269744:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077727.269749:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077727.269761:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077727.269783:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923898496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077727.269793:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077727.269803:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077727.269805:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077727.269809:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4416 00000100:00080000:1.0:1769077727.269813:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077727.269838:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077727.269844:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077727.269863:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923898624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077727.269872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077727.269885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077727.269887:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077727.269892:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4416 00000100:00080000:1.0:1769077727.269897:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077727.269901:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769077727.269906:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:1.0:1769077727.269913:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077727.269927:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923898752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077727.269932:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923898368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077727.269953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077727.269957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077727.269959:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077727.269964:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4416 00000100:00080000:1.0:1769077727.269969:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077727.269972:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077727.269978:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077727.270025:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923898880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077727.270026:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077727.270070:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077727/real 1769077727] req@ffff969e10b60700 x1855007923898368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077727.270092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923898368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077727.270101:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077727.270105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077727.270109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923898368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077727.270117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077727.270141:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077727.270153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077727/real 1769077727] req@ffff969e10b62d80 x1855007923898496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077727.270166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923898496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077727.270174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077727.270177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077727.270180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077727.270187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077727.270205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077727.270215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077727/real 1769077727] req@ffff969e10b60000 x1855007923898624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077727.270240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923898624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077727.270248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077727.270251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077727.270254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077727.270281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077727.270300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077727.270310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077727/real 1769077727] req@ffff969e10b61180 x1855007923898752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077727.270341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923898752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077727.270348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077727.270352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077727.270355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077727.270361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077727.270378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077727.270388:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077727/real 1769077727] req@ffff969e10b60380 x1855007923898880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077727.270398:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923898880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077727.270418:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077727.270421:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077727.270425:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923898880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077727.588706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077728.292838:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077728.612727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077729.317153:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077729.636734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077730.340792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077730.660705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077731.364779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077731.684676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077732.388716:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077732.388727:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077732.388732:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077732.388743:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4421 00000100:00080000:1.0:1769077732.388754:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077732.388761:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077732.388800:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923899008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077732.388830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077732.388834:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077732.388836:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077732.388840:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4421 00000100:00080000:1.0:1769077732.388844:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077732.388848:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077732.388858:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077732.388872:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923899136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077732.388883:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077732.388886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077732.388888:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077732.388892:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4421 00000100:00080000:1.0:1769077732.388898:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077732.388911:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923899008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077732.388919:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077732.388938:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077732.388960:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923899264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077732.388972:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077732.388975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077732.388977:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077732.388980:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4421 00000100:00080000:1.0:1769077732.388986:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077732.388991:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077732.388997:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077732.389005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077732.389019:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923899392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077732.389032:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077732.389036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077732.389039:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077732.389046:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4421 00000100:00080000:1.0:1769077732.389052:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077732.389055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077732/real 1769077732] req@ffff969e10b60380 x1855007923899008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077732.389063:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077732.389076:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077732.389080:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923899008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077732.389095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1769077732.389096:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923899520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077732.389102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077732.389118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923899008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077732.389128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077732.389151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077732.389163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077732/real 1769077732] req@ffff969e10b61180 x1855007923899136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077732.389174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923899136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077732.389179:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077732.389195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077732.389198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899136:192.168.204.143@tcp:38:kworker.0 00010000:00010000:3.0:1769077732.389222:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077732.389263:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077732.389291:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077732.389302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077732/real 1769077732] req@ffff969e10b60000 x1855007923899264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077732.389315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923899264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077732.389323:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077732.389326:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077732.389329:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077732.389336:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077732.389356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077732.389366:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077732/real 1769077732] req@ffff969e10b62d80 x1855007923899392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077732.389377:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923899392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077732.389383:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077732.389385:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077732.389387:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077732.389391:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077732.389410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077732.389416:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077732/real 1769077732] req@ffff969e10b60700 x1855007923899520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077732.389423:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923899520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077732.389429:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077732.389431:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077732.389433:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077732.708713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077733.412759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077733.732659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077734.436747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077734.756703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077735.460802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077735.780720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077736.484825:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077736.804758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077737.508735:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077737.508747:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077737.508752:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077737.508762:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4426 00000100:00080000:0.0:1769077737.508774:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077737.508782:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077737.508816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077737.508828:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923899648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077737.508855:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077737.508859:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077737.508861:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077737.508867:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4426 00000100:00080000:0.0:1769077737.508871:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077737.508875:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077737.508887:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077737.508901:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923899776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077737.508915:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923899648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077737.508934:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077737.508937:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077737.508939:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077737.508943:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4426 00000100:00080000:0.0:1769077737.508946:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077737.508951:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077737.508955:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077737.508969:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923899904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077737.508977:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077737.508981:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077737.508983:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077737.508987:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4426 00000800:00000100:2.0:1769077737.508987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077737.508990:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077737.508993:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077737.508998:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077737.509007:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923900032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077737.509016:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077737.509019:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077737.509021:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077737.509023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077737/real 1769077737] req@ffff969e06a61880 x1855007923899648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077737.509025:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4426 00000100:00080000:0.0:1769077737.509034:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077737.509038:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077737.509041:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923899648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077737.509044:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769077737.509052:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077737.509054:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923900160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077737.509057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077737.509062:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923899648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077737.509105:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077737.509130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077737.509173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077737/real 1769077737] req@ffff969e06a63100 x1855007923899776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077737.509194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923899776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077737.509200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077737.509203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077737.509206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077737.509212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077737.509233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077737.509245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077737/real 1769077737] req@ffff969e06a61500 x1855007923899904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077737.509258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923899904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077737.509268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077737.509271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077737.509275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923899904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077737.509281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077737.509297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077737.509304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077737/real 1769077737] req@ffff969e06a60380 x1855007923900032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077737.509312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923900032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077737.509326:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077737.509329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077737.509331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077737.509335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077737.509348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077737.509354:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077737/real 1769077737] req@ffff969e06a61f80 x1855007923900160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077737.509361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923900160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077737.509367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077737.509369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077737.509371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077737.828673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077738.532786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077738.852733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077739.556737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077739.876719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077740.580781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077740.900748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077741.604795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077741.924656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077742.628707:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077742.628723:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077742.628729:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077742.628743:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4431 00000100:00080000:0.0:1769077742.628756:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077742.628766:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077742.628792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077742.628826:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923900288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077742.628857:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077742.628861:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077742.628863:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077742.628868:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4431 00000100:00080000:0.0:1769077742.628873:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077742.628884:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077742.628897:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077742.628912:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923900416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077742.628919:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923900288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077742.628923:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077742.628947:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077742.628949:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077742.628952:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4431 00000100:00080000:0.0:1769077742.628956:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077742.628960:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077742.628964:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077742.628974:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923900544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077742.628984:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077742.628987:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077742.628990:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077742.628994:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4431 00000100:00080000:0.0:1769077742.628999:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077742.629003:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:2.0:1769077742.629008:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769077742.629013:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077742.629034:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923900672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077742.629044:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077742.629048:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1769077742.629066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077742/real 1769077742] req@ffff969e06a62680 x1855007923900288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077742.629067:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077742.629074:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4431 00000100:00080000:0.0:1769077742.629080:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077742.629088:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077742.629101:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077742.629102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923900288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077742.629119:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077742.629120:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923900800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077742.629134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077742.629142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923900288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077742.629176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077742.629198:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077742.629209:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077742/real 1769077742] req@ffff969e06a61180 x1855007923900416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077742.629218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923900416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077742.629224:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077742.629226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077742.629228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077742.629233:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077742.629243:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077742.629249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077742/real 1769077742] req@ffff969e06a62d80 x1855007923900544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077742.629257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923900544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077742.629265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077742.629268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077742.629271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077742.629278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077742.629299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077742.629310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077742/real 1769077742] req@ffff969e06a60700 x1855007923900672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077742.629331:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923900672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077742.629342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077742.629345:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077742.629349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077742.629356:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077742.629377:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077742.629387:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077742/real 1769077742] req@ffff969e06a63800 x1855007923900800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077742.629400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923900800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077742.629410:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077742.629414:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077742.629418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923900800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077742.948735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077743.652802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077743.972688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077744.676755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077744.996734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077745.700778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077746.020669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077746.724741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077747.044704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077747.748782:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077747.748794:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077747.748800:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769077747.748805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077747.748815:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4436 00000100:00080000:2.0:1769077747.748830:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077747.748838:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077747.748878:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923900928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077747.748928:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077747.748932:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077747.748935:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077747.748941:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4436 00000100:00080000:2.0:1769077747.748947:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077747.748953:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077747.748963:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923900928:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077747.748963:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077747.749001:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923901056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077747.749015:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077747.749019:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077747.749021:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077747.749026:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4436 00000100:00080000:2.0:1769077747.749032:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077747.749036:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077747.749042:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077747.749054:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077747.749059:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923901184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077747.749072:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077747.749075:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077747.749078:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077747.749082:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4436 00000100:00100000:0.0:1769077747.749089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077747/real 1769077747] req@ffff969e0a46ed80 x1855007923900928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077747.749101:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077747.749109:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077747.749116:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077747.749119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923900928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077747.749129:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077747.749134:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077747.749134:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923901312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077747.749141:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923900928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077747.749167:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077747.749169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901056:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077747.749176:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077747.749180:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077747.749186:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4436 00000100:00080000:2.0:1769077747.749191:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077747.749192:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077747.749200:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077747.749205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077747/real 1769077747] req@ffff969e0a46e300 x1855007923901056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769077747.749209:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077747.749220:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923901056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077747.749227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077747.749229:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923901440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077747.749232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077747.749239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077747.749248:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077747.749273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077747.749282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077747/real 1769077747] req@ffff969e0a46f100 x1855007923901184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077747.749291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923901184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077747.749296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077747.749298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077747.749300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077747.749304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077747.749315:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077747.749320:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077747/real 1769077747] req@ffff969e0a46df80 x1855007923901312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077747.749328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923901312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077747.749333:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077747.749335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077747.749337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077747.749348:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077747.749358:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077747.749363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077747/real 1769077747] req@ffff969e0a46ca80 x1855007923901440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077747.749380:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923901440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077747.749386:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077747.749389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077747.749391:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077748.068744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077748.772749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077749.092731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077749.796818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077750.116732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077750.820821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077751.140714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077751.844822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077752.164711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077752.868804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077752.868814:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077752.868818:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077752.868828:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4442 00010000:00010000:3.0:1769077752.868828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077752.868841:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077752.868847:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077752.868896:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923901568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077752.868918:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077752.868921:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077752.868937:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077752.868941:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4442 00000100:00080000:1.0:1769077752.868945:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077752.868948:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077752.868958:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077752.868968:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923901696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077752.868977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077752.868979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923901568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077752.868981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077752.868984:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077752.868993:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4442 00000100:00080000:1.0:1769077752.868995:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077752.868998:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077752.869001:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077752.869010:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923901824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077752.869016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077752.869019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077752.869020:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077752.869023:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4442 00000100:00080000:1.0:1769077752.869025:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077752.869027:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077752.869030:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077752.869045:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923901952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077752.869052:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077752.869054:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077752.869055:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077752.869058:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4442 00000800:00000100:0.0:1769077752.869063:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769077752.869065:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077752.869070:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077752.869074:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077752.869082:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923902080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077752.869099:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077752/real 1769077752] req@ffff969e10b62680 x1855007923901568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077752.869125:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923901568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077752.869137:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077752.869142:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077752.869147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923901568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077752.869171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077752.869199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077752.869211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077752/real 1769077752] req@ffff969e10b62a00 x1855007923901696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077752.869225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923901696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077752.869249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077752.869253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077752.869258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077752.869274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077752.869299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077752.869310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077752/real 1769077752] req@ffff969e10b60a80 x1855007923901824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077752.869323:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923901824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077752.869332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077752.869336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077752.869340:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077752.869347:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077752.869368:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077752.869380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077752/real 1769077752] req@ffff969e10b61f80 x1855007923901952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077752.869393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923901952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077752.869403:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077752.869406:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077752.869427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923901952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077752.869435:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077752.869461:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077752.869481:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077752/real 1769077752] req@ffff969e10b61500 x1855007923902080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077752.869495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923902080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077752.869504:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077752.869508:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077752.869512:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077753.188666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077753.892801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077754.212677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077754.916805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077755.236692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077755.940771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077756.260730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077756.964847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077757.284686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077757.988732:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077757.988742:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077757.988746:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077757.988756:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4447 00000100:00080000:3.0:1769077757.988766:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077757.988785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077757.988812:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077757.988844:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923902208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077757.988865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077757.988868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077757.988869:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077757.988873:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4447 00000100:00080000:3.0:1769077757.988877:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077757.988880:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077757.988889:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077757.988896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923902208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077757.988901:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923902336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077757.988911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077757.988913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077757.988914:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077757.988916:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4447 00000100:00080000:3.0:1769077757.988919:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077757.988921:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077757.988925:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077757.988933:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923902464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077757.988939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077757.988941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077757.988949:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077757.988952:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4447 00000100:00080000:3.0:1769077757.988956:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077757.988959:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077757.988963:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769077757.988967:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077757.988976:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923902592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077757.988984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077757.988986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077757.988988:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077757.988990:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4447 00000100:00080000:3.0:1769077757.988993:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077757.988995:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077757.988999:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077757.989003:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077757/real 1769077757] req@ffff969e0a7fdf80 x1855007923902208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077757.989007:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923902720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077757.989035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923902208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077757.989044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077757.989048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077757.989059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923902208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077757.989078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077757.989095:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077757.989103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077757/real 1769077757] req@ffff969e0a7fca80 x1855007923902336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077757.989111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923902336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077757.989116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077757.989118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077757.989120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077757.989124:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077757.989134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077757.989140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077757/real 1769077757] req@ffff969e0a7fe300 x1855007923902464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077757.989147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923902464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077757.989152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077757.989154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077757.989156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077757.989159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077757.989169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077757.989181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077757/real 1769077757] req@ffff969e0a7fd180 x1855007923902592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077757.989189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923902592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077757.989194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077757.989196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077757.989198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077757.989201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077757.989212:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077757.989218:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077757/real 1769077757] req@ffff969e0a7fd880 x1855007923902720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077757.989225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923902720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077757.989231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077757.989232:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077757.989234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077758.308716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077759.012826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077759.332678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077760.036817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077760.356724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077761.060787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077761.380671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077762.084836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077762.404783:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077763.108757:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077763.108766:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077763.108770:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077763.108779:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4452 00000100:00080000:3.0:1769077763.108789:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077763.108795:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077763.108829:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923902848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769077763.108833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077763.108871:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077763.108874:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077763.108876:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077763.108879:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4452 00000100:00080000:3.0:1769077763.108882:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077763.108885:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077763.108894:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077763.108904:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923902976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077763.108913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077763.108914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923902848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077763.108916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077763.108919:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077763.108927:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4452 00000100:00080000:3.0:1769077763.108930:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077763.108932:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077763.108935:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077763.108944:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923903104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077763.108950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077763.108952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077763.108954:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077763.108956:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4452 00000100:00080000:3.0:1769077763.108964:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077763.108967:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077763.108971:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077763.108979:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923903232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077763.108986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077763.108988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077763.108989:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077763.108991:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4452 00000100:00080000:3.0:1769077763.108993:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077763.108996:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077763.108998:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077763.109005:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923903360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077763.109038:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077763.109087:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077763.109125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077763.109152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077763.428648:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077764.132788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077764.452720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077765.156832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077765.476751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077766.180715:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077766.180829:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769077766.180864:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077766.180879:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077766.180923:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077766.180932:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077766.180940:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077766.180948:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:0.0:1769077766.181028:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077763/real 1769077766] req@ffff969e0a7fd500 x1855007923902976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077766.181056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923902976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077766.181071:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077766.181077:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077766.181083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923902976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077766.181091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077763/real 1769077766] req@ffff969e0a7fdc00 x1855007923903104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077766.181105:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923903104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077766.181115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077766.181119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077766.181123:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077766.181130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077763/real 1769077766] req@ffff969e0a7ff800 x1855007923903232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077766.181171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923903232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077766.181181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077766.181186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077766.181190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077766.181198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077763/real 1769077766] req@ffff969e0a7fc000 x1855007923903360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077766.181212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923903360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077766.181221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077766.181225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077766.181229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077766.181237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077763/real 1769077766] req@ffff969e0a7fc380 x1855007923902848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077766.181251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923902848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077766.181277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077766.181282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077766.181287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923902848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077766.500674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077767.204850:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077767.524734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077768.228763:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077768.228789:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077768.228793:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077768.228803:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4457 00000100:00080000:0.0:1769077768.228814:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077768.228822:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077768.228841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077768.228860:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923903488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077768.228888:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077768.228892:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077768.228893:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077768.228898:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4457 00000100:00080000:0.0:1769077768.228903:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077768.228907:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077768.228917:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077768.228928:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923903488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077768.228932:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923903616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077768.228946:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077768.228950:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077768.228952:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077768.228956:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4457 00000100:00080000:0.0:1769077768.228961:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077768.228968:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077768.228977:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077768.228992:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923903744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077768.229010:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077768.229015:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077768.229018:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077768.229020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077768.229030:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4457 00000100:00080000:0.0:1769077768.229038:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077768.229044:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077768.229051:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077768.229068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077768/real 1769077768] req@ffff969e06a60000 x1855007923903488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077768.229069:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923903872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077768.229104:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077768.229109:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077768.229112:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077768.229115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923903488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077768.229120:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4457 00000100:00080000:0.0:1769077768.229128:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077768.229134:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077768.229140:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769077768.229157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077768.229160:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923904000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077768.229167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077768.229177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923903488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077768.229209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077768.229237:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077768.229251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077768/real 1769077768] req@ffff969e06a60a80 x1855007923903616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077768.229263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923903616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077768.229272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077768.229275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077768.229278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077768.229284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077768.229301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077768.229310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077768/real 1769077768] req@ffff969e06a63800 x1855007923903744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077768.229321:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923903744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077768.229329:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077768.229332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077768.229335:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077768.229349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077768.229371:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077768.229381:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077768/real 1769077768] req@ffff969e06a60700 x1855007923903872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077768.229393:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923903872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077768.229401:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077768.229405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077768.229407:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923903872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077768.229413:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077768.229429:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077768.229438:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077768/real 1769077768] req@ffff969e06a62d80 x1855007923904000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077768.229448:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923904000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077768.229455:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077768.229459:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077768.229462:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077768.548717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077769.252847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077769.572672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077770.276775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077770.596658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077771.300792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077771.620716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077772.324834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077772.644658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077773.348708:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077773.348719:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077773.348723:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077773.348732:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4462 00000100:00080000:0.0:1769077773.348761:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077773.348768:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077773.348802:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923904128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769077773.348815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077773.348833:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077773.348837:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077773.348839:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077773.348842:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4462 00000100:00080000:0.0:1769077773.348845:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077773.348849:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077773.348858:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077773.348869:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923904256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077773.348878:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077773.348884:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769077773.348886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923904128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077773.348895:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077773.348899:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4462 00000100:00080000:0.0:1769077773.348902:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077773.348905:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077773.348910:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077773.348934:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923904384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077773.348942:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077773.348954:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077773.348957:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077773.348961:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4462 00000100:00080000:0.0:1769077773.348965:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077773.348968:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077773.348972:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769077773.348975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077773.348995:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923904512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077773.349011:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769077773.349020:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077773/real 1769077773] req@ffff969e06a61180 x1855007923904128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077773.349033:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077773.349040:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077773.349048:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4462 00000100:00100000:1.0:1769077773.349051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923904128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077773.349055:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077773.349069:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769077773.349074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077773.349076:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769077773.349082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077773.349088:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923904128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077773.349093:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923904640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077773.349127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077773.349155:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077773.349179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077773/real 1769077773] req@ffff969e06a62680 x1855007923904256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077773.349192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923904256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077773.349201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077773.349204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077773.349208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077773.349223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077773.349264:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077773.349277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077773/real 1769077773] req@ffff969e06a61f80 x1855007923904384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077773.349286:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923904384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077773.349291:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077773.349294:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077773.349296:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077773.349301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077773.349316:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077773.349323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077773/real 1769077773] req@ffff969e06a60380 x1855007923904512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077773.349337:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923904512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077773.349344:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077773.349347:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077773.349349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077773.349353:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077773.349365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077773.349371:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077773/real 1769077773] req@ffff969e06a61500 x1855007923904640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077773.349389:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923904640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077773.349396:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077773.349398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077773.349400:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077773.668731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077774.372788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077774.692698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077775.396793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077775.716721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077776.420804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077776.740728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077777.444788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077777.764709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077778.468684:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077778.468694:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077778.468699:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077778.468708:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4467 00000100:00080000:0.0:1769077778.468718:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077778.468725:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077778.468742:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077778.468760:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923904768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077778.468797:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077778.468800:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077778.468801:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077778.468805:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4467 00000100:00080000:0.0:1769077778.468809:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077778.468812:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077778.468821:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077778.468828:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923904768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077778.468834:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923904896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077778.468842:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077778.468845:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077778.468847:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077778.468849:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4467 00000100:00080000:0.0:1769077778.468852:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077778.468855:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077778.468858:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077778.468867:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923905024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077778.468873:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077778.468876:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077778.468877:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077778.468880:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4467 00000100:00080000:0.0:1769077778.468888:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077778.468892:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769077778.468897:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769077778.468900:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077778.468911:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923905152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077778.468918:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077778.468920:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077778.468922:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077778.468925:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4467 00000100:00080000:0.0:1769077778.468927:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769077778.468929:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077778/real 1769077778] req@ffff969e06a63100 x1855007923904768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077778.468933:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077778.468940:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077778.468949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923905280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769077778.468953:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923904768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077778.468966:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077778.468970:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077778.468974:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923904768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077778.468993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077778.469009:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077778.469017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077778/real 1769077778] req@ffff969e06a61880 x1855007923904896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077778.469038:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923904896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077778.469049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077778.469053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077778.469057:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923904896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077778.469064:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077778.469087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077778.469097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077778/real 1769077778] req@ffff969e06a61500 x1855007923905024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077778.469106:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923905024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077778.469112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077778.469115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077778.469117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077778.469121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077778.469134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077778.469140:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077778/real 1769077778] req@ffff969e06a60380 x1855007923905152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077778.469148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923905152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077778.469163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077778.469166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077778.469169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077778.469173:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077778.469186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077778.469206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077778/real 1769077778] req@ffff969e06a61f80 x1855007923905280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077778.469214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923905280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077778.469220:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077778.469222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077778.469224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077778.788663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077779.492793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077779.812718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077780.516787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077780.836694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077781.540768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077781.860718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077782.564809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077782.884653:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077783.588718:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077783.588728:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077783.588732:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769077783.588736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077783.588745:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4472 00000100:00080000:1.0:1769077783.588757:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077783.588766:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077783.588820:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923905408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077783.588850:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077783.588854:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077783.588856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077783.588861:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4472 00000100:00080000:1.0:1769077783.588866:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077783.588871:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077783.588881:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077783.588895:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923905536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077783.588903:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077783.588906:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077783.588908:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077783.588914:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923905408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077783.588915:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4472 00000100:00080000:1.0:1769077783.588922:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077783.588927:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077783.588931:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077783.588942:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923905664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077783.588949:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077783.588952:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077783.588954:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077783.588962:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4472 00000100:00080000:1.0:1769077783.588966:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077783.588969:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077783.588973:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077783.588984:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923905792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077783.588985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769077783.588995:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077783.588998:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077783.588999:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077783.589002:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4472 00000100:00080000:1.0:1769077783.589005:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077783.589007:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077783.589011:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077783.589027:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923905920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077783.589031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077783/real 1769077783] req@ffff969e10b61500 x1855007923905408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077783.589058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923905408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077783.589069:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077783.589074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077783.589078:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923905408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077783.589100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077783.589119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077783.589128:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077783/real 1769077783] req@ffff969e10b61f80 x1855007923905536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077783.589137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923905536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077783.589149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077783.589152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077783.589155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077783.589159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077783.589173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077783.589179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077783/real 1769077783] req@ffff969e10b60a80 x1855007923905664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077783.589187:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923905664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077783.589193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077783.589203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077783.589205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077783.589210:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077783.589222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077783.589232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077783/real 1769077783] req@ffff969e10b62a00 x1855007923905792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077783.589244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923905792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077783.589253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077783.589257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077783.589267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077783.589274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077783.589297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077783.589307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077783/real 1769077783] req@ffff969e10b62680 x1855007923905920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077783.589316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923905920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077783.589322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077783.589324:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077783.589327:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923905920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077783.908718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077784.612785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077784.932739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077785.636773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077785.956709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077786.660806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077786.980728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077787.684717:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077788.004652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077788.708735:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077788.708746:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077788.708751:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077788.708760:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4477 00010000:00010000:0.0:1769077788.708768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077788.708771:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077788.708780:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077788.708815:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923906048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077788.708840:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077788.708860:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077788.708862:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077788.708865:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4477 00000100:00080000:3.0:1769077788.708869:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077788.708875:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077788.708878:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923906048:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077788.708884:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077788.708899:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923906176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077788.708912:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077788.708916:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077788.708918:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077788.708923:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4477 00000100:00080000:3.0:1769077788.708928:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077788.708933:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077788.708939:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077788.708953:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923906304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769077788.708957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077788.708968:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077788.708971:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077788.708973:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077788.708977:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4477 00000100:00080000:3.0:1769077788.708981:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077788.708984:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077788.708994:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077788.709003:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077788/real 1769077788] req@ffff969e0a7ffb80 x1855007923906048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077788.709006:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923906432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077788.709028:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077788.709031:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077788.709034:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077788.709040:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4477 00000100:00080000:3.0:1769077788.709046:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077788.709051:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077788.709054:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077788.709062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923906048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077788.709064:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923906560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077788.709088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077788.709094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077788.709098:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923906048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077788.709118:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077788.709137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077788.709145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077788/real 1769077788] req@ffff969e0a7fc380 x1855007923906176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077788.709153:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923906176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077788.709159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077788.709161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077788.709163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077788.709167:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769077788.709177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077788.709182:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077788/real 1769077788] req@ffff969e0a7fc000 x1855007923906304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077788.709189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923906304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077788.709204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077788.709206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077788.709209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769077788.709212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077788.709227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077788.709233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077788/real 1769077788] req@ffff969e0a7ff800 x1855007923906432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077788.709240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923906432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077788.709245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077788.709247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077788.709249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077788.709252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769077788.709263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769077788.709272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077788/real 1769077788] req@ffff969e0a7fdc00 x1855007923906560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077788.709317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923906560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077788.709327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769077788.709330:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769077788.709339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077789.028727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077789.732754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077790.052728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077790.757108:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077791.076758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077791.780877:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077792.100715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077792.804970:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077793.124726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077793.828789:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077793.828807:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077793.828813:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077793.828842:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4483 00000100:00080000:2.0:1769077793.828857:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077793.828866:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077793.828934:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923906688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077793.828960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077793.828964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077793.828966:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077793.828971:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4483 00000100:00080000:2.0:1769077793.828975:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077793.828979:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077793.828992:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077793.829006:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923906816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077793.829016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077793.829020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077793.829021:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077793.829025:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4483 00000100:00080000:2.0:1769077793.829029:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077793.829033:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077793.829038:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077793.829050:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923906944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077793.829074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077793.829077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077793.829080:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077793.829084:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4483 00000100:00080000:2.0:1769077793.829088:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077793.829092:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077793.829098:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077793.829110:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923907072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077793.829121:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077793.829124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077793.829126:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077793.829130:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4483 00000100:00080000:2.0:1769077793.829134:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077793.829138:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077793.829142:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077793.829150:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923907200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077793.829800:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923906688:192.168.204.143@tcp:250:kworker.0 00010000:00010000:0.0:1769077793.829832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1769077793.829902:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077793.829943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077793/real 1769077793] req@ffff969e0a46ce00 x1855007923906688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077793.829973:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923906688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077793.829981:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077793.829985:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077793.829988:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923906688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077793.829995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077793.830018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077793.830032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077793/real 1769077793] req@ffff969e0a46f800 x1855007923906816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077793.830046:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923906816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077793.830054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077793.830057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077793.830061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077793.830068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077793.830100:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077793.830109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077793/real 1769077793] req@ffff969e0a46f480 x1855007923906944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077793.830134:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923906944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077793.830150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077793.830153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077793.830156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923906944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077793.830161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077793.830182:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077793.830224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077793/real 1769077793] req@ffff969e0a46ea00 x1855007923907072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077793.830238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923907072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077793.830248:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077793.830251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077793.830255:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077793.830262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077793.830283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077793.830300:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077793/real 1769077793] req@ffff969e0a46d500 x1855007923907200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077793.830312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923907200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077793.830318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077793.830320:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077793.830330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077794.148691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077794.853007:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077795.172705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077795.876955:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077796.196696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077796.900761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077797.220700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077797.924780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077798.244654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077798.948683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077798.948692:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077798.948696:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077798.948705:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4488 00000100:00080000:2.0:1769077798.948716:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077798.948722:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077798.948762:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923907328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769077798.948763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077798.948797:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077798.948801:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077798.948803:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077798.948807:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4488 00000100:00080000:2.0:1769077798.948810:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077798.948813:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077798.948822:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077798.948841:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923907456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077798.948850:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077798.948853:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077798.948856:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077798.948857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923907328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077798.948860:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4488 00000100:00080000:2.0:1769077798.948866:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077798.948884:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077798.948889:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077798.948898:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923907584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077798.948906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077798.948909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077798.948910:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077798.948913:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4488 00000100:00080000:2.0:1769077798.948937:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077798.948941:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077798.948948:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:3.0:1769077798.948949:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077798.948963:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923907712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077798.948981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077798.948983:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077798.948985:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077798.948989:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4488 00000100:00080000:2.0:1769077798.948993:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077798.948994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077798/real 1769077798] req@ffff969e0a46d500 x1855007923907328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077798.948999:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077798.949006:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077798.949019:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923907840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077798.949028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923907328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077798.949042:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077798.949047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077798.949052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923907328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077798.949078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077798.949148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077798.949164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077798/real 1769077798] req@ffff969e0a46ea00 x1855007923907456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077798.949190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923907456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077798.949200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077798.949205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077798.949208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077798.949216:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077798.949238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077798.949249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077798/real 1769077798] req@ffff969e0a46f480 x1855007923907584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077798.949260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923907584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077798.949268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077798.949271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077798.949275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077798.949281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077798.949300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077798.949312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077798/real 1769077798] req@ffff969e0a46f800 x1855007923907712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077798.949339:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923907712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077798.949349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077798.949353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077798.949356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077798.949363:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077798.949385:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077798.949397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077798/real 1769077798] req@ffff969e0a46ce00 x1855007923907840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077798.949409:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923907840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077798.949419:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077798.949423:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077798.949426:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923907840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077799.268676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077799.972847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077800.292656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077800.996720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077801.316727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077802.020766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077802.340727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077803.044804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077803.364701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077804.068739:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077804.068759:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077804.068764:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077804.068775:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4493 00000100:00080000:3.0:1769077804.068800:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077804.068807:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077804.068813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077804.068874:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923907968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077804.068924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077804.068927:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077804.068929:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077804.068933:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4493 00000100:00080000:3.0:1769077804.068937:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077804.068941:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077804.068950:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077804.068962:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923908096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077804.068986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077804.068989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077804.068990:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077804.068992:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4493 00000100:00080000:3.0:1769077804.068994:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077804.068996:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077804.069000:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077804.069007:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923908224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077804.069014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077804.069017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077804.069018:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077804.069033:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4493 00000100:00080000:3.0:1769077804.069041:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077804.069043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923907968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077804.069047:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077804.069052:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077804.069066:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923908352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077804.069084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077804.069091:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077804.069093:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077804.069095:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4493 00000100:00080000:3.0:1769077804.069098:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077804.069101:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077804.069105:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077804.069113:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923908480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077804.069140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077804.069175:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077804/real 1769077804] req@ffff969e0a7fd500 x1855007923907968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077804.069189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923907968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077804.069197:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077804.069201:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077804.069204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923907968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077804.069209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077804.069226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077804.069233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077804/real 1769077804] req@ffff969e0a7fd880 x1855007923908096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077804.069241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923908096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077804.069247:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077804.069249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077804.069251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077804.069254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077804.069265:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077804.069280:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077804/real 1769077804] req@ffff969e0a7fd180 x1855007923908224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077804.069298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923908224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077804.069304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077804.069306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077804.069308:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077804.069327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077804.069340:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077804.069346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077804/real 1769077804] req@ffff969e0a7fe300 x1855007923908352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077804.069358:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923908352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077804.069367:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077804.069370:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077804.069374:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077804.069380:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077804.069399:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077804.069410:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077804/real 1769077804] req@ffff969e0a7fca80 x1855007923908480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077804.069421:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923908480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077804.069441:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077804.069446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077804.069449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077804.388705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077805.092804:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077805.412707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077806.116843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077806.436716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077807.140869:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077807.460736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077808.164809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077808.484703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077809.188709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077809.188719:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077809.188724:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077809.188732:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4498 00000100:00080000:2.0:1769077809.188743:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077809.188744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077809.188750:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077809.188784:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923908608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077809.188809:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077809.188812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077809.188813:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077809.188816:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4498 00000100:00080000:2.0:1769077809.188819:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077809.188822:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077809.188830:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077809.188842:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923908736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077809.188844:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923908608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077809.188858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077809.188861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077809.188879:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077809.188883:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4498 00000100:00080000:2.0:1769077809.188901:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077809.188904:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077809.188908:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077809.188910:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077809.188922:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923908864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077809.188931:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077809.188933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077809.188945:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077809.188949:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4498 00000100:00080000:2.0:1769077809.188952:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077809.188957:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077809.188961:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077809.188962:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077809/real 1769077809] req@ffff969e0a46d880 x1855007923908608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077809.188970:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923908992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077809.188979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077809.188982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077809.188983:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077809.188987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923908608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077809.188989:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4498 00000100:00080000:2.0:1769077809.188993:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077809.188999:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077809.189004:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769077809.189007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077809.189013:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077809.189018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923908608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077809.189021:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923909120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077809.189056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077809.189075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077809.189084:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077809/real 1769077809] req@ffff969e0a46c380 x1855007923908736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077809.189093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923908736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077809.189099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077809.189101:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077809.189104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077809.189108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077809.189118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077809.189124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077809/real 1769077809] req@ffff969e0a46ce00 x1855007923908864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077809.189131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923908864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077809.189136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077809.189138:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077809.189140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077809.189143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077809.189153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077809.189159:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077809/real 1769077809] req@ffff969e0a46f800 x1855007923908992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077809.189172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923908992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077809.189181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077809.189184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077809.189188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923908992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077809.189194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077809.189215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077809.189226:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077809/real 1769077809] req@ffff969e0a46f480 x1855007923909120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077809.189239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923909120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077809.189249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077809.189253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077809.189256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077809.508705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077810.212789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077810.532737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077811.236761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077811.556692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077812.260772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077812.580654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077813.284788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077813.604656:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077814.308704:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077814.308717:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077814.308722:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077814.308733:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4503 00000100:00080000:2.0:1769077814.308743:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077814.308749:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077814.308778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077814.308784:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923909248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077814.308810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077814.308813:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077814.308814:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077814.308817:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4503 00000100:00080000:2.0:1769077814.308820:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077814.308823:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077814.308830:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077814.308839:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923909376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077814.308845:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077814.308860:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077814.308863:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077814.308868:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4503 00000100:00080000:2.0:1769077814.308872:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077814.308877:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923909248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077814.308892:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077814.308897:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077814.308909:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923909504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077814.308917:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077814.308919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077814.308921:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077814.308924:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4503 00000100:00080000:2.0:1769077814.308938:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077814.308941:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077814.308944:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:3.0:1769077814.308950:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077814.308955:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923909632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077814.308963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077814.308965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077814.308966:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077814.308968:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4503 00000100:00080000:2.0:1769077814.308971:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077814.308973:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077814.308976:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077814.308982:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923909760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077814.308999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077814/real 1769077814] req@ffff969e0a46ea00 x1855007923909248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077814.309022:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923909248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077814.309048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077814.309054:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077814.309059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923909248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077814.309067:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077814.309088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077814.309107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077814/real 1769077814] req@ffff969e0a46d500 x1855007923909376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077814.309117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923909376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077814.309123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077814.309125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077814.309128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077814.309152:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077814.309173:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077814.309185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077814/real 1769077814] req@ffff969e0a46ca80 x1855007923909504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077814.309199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923909504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077814.309208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077814.309212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077814.309214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077814.309225:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077814.309247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077814.309258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077814/real 1769077814] req@ffff969e0a46df80 x1855007923909632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077814.309288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923909632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077814.309298:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077814.309302:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077814.309306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077814.309313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077814.309334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077814.309346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077814/real 1769077814] req@ffff969e0a46f100 x1855007923909760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077814.309358:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923909760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077814.309364:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077814.309367:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077814.309369:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923909760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077814.628709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077815.332801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077815.652721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077816.356777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077816.676723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077817.380795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077817.700686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077818.404768:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077818.724717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077819.428750:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077819.428760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077819.428776:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077819.428801:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4508 00000100:00080000:3.0:1769077819.428811:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1769077819.428818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077819.428818:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077819.428850:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923909888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077819.428876:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077819.428878:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077819.428880:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077819.428883:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4508 00000100:00080000:3.0:1769077819.428886:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077819.428889:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077819.428897:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077819.428907:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923910016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077819.428915:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077819.428918:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077819.428919:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077819.428924:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4508 00000100:00080000:3.0:1769077819.428930:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077819.428932:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923909888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077819.428935:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077819.428939:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077819.428945:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923910144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077819.428978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077819.428981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077819.428983:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077819.428987:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4508 00000100:00080000:3.0:1769077819.428991:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077819.428994:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077819.428998:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077819.429006:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923910272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077819.429012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077819.429015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077819.429016:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077819.429019:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4508 00000100:00080000:3.0:1769077819.429037:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077819.429040:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077819.429043:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077819.429052:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923910400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077819.429054:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077819.429111:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077819/real 1769077819] req@ffff969e0a7fdf80 x1855007923909888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077819.429136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923909888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077819.429150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077819.429155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077819.429160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923909888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077819.429189:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077819.429211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077819.429220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077819/real 1769077819] req@ffff969e0a7fea00 x1855007923910016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077819.429229:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923910016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077819.429236:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077819.429238:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077819.429241:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077819.429246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077819.429257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077819.429263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077819/real 1769077819] req@ffff969e0a7fc700 x1855007923910144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077819.429282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923910144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077819.429288:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077819.429290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077819.429292:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077819.429297:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077819.429310:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077819.429317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077819/real 1769077819] req@ffff969e0a7fce00 x1855007923910272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077819.429325:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923910272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077819.429330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077819.429332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077819.429334:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077819.429338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077819.429348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077819.429353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077819/real 1769077819] req@ffff969e0a7fe680 x1855007923910400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077819.429360:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923910400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077819.429375:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077819.429377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077819.429380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077819.748662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077820.452731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077820.772704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077821.476821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077821.796651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077822.500722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077822.820700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077823.524764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077823.844750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077824.548741:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077824.548751:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077824.548756:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077824.548766:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4513 00000100:00080000:0.0:1769077824.548778:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077824.548786:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077824.548788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077824.548838:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923910528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077824.548876:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077824.548879:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077824.548881:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077824.548884:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4513 00000100:00080000:0.0:1769077824.548888:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077824.548892:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077824.548901:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077824.548912:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923910656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077824.548927:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077824.548930:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077824.548932:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077824.548935:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4513 00000100:00080000:0.0:1769077824.548939:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077824.548941:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077824.548945:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077824.548949:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923910528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077824.548956:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923910784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077824.548968:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077824.548972:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077824.548975:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077824.548979:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4513 00000100:00080000:0.0:1769077824.548990:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077824.548995:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077824.549001:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077824.549016:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923910912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077824.549026:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077824.549030:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077824.549032:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077824.549038:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4513 00000800:00000100:2.0:1769077824.549040:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077824.549045:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077824.549062:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077824.549069:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077824.549081:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923911040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077824.549086:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077824/real 1769077824] req@ffff969e06a62680 x1855007923910528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077824.549109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923910528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077824.549121:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077824.549126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077824.549130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923910528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077824.549147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077824.549174:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077824.549187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077824/real 1769077824] req@ffff969e06a61180 x1855007923910656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077824.549201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923910656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077824.549211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077824.549215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077824.549219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077824.549254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077824.549277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077824.549290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077824/real 1769077824] req@ffff969e06a62d80 x1855007923910784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077824.549303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923910784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077824.549311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077824.549315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077824.549319:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077824.549326:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077824.549346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077824.549357:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077824/real 1769077824] req@ffff969e06a60700 x1855007923910912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077824.549369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923910912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077824.549378:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077824.549382:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077824.549386:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923910912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077824.549392:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077824.549408:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077824.549416:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077824/real 1769077824] req@ffff969e06a63800 x1855007923911040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077824.549430:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923911040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077824.549436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077824.549438:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077824.549440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077824.868723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077825.572813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077825.892745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077826.596749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077826.916762:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077827.620784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077827.940740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077828.644806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077828.964718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077829.668739:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077829.668751:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1769077829.668756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077829.668756:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077829.668769:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4518 00000100:00080000:3.0:1769077829.668781:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077829.668789:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077829.668827:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923911168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077829.668857:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077829.668862:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077829.668864:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077829.668869:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4518 00000100:00080000:3.0:1769077829.668874:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077829.668879:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077829.668889:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077829.668900:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923911168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077829.668919:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923911296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077829.668932:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077829.668936:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077829.668938:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077829.668942:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4518 00000100:00080000:3.0:1769077829.668954:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077829.668960:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077829.668966:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077829.668991:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077829.668997:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923911424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077829.669012:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077829.669015:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077829.669017:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077829.669022:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4518 00000100:00080000:3.0:1769077829.669027:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077829.669032:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077829.669038:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911424:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769077829.669038:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077829.669056:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923911552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077829.669067:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077829.669071:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077829.669073:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077829.669081:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911552:192.168.204.143@tcp:8:kworker.0 00000100:00080000:3.0:1769077829.669081:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4518 00000100:00080000:3.0:1769077829.669089:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077829.669095:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077829.669100:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077829.669112:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923911680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077829.669147:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077829.988731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077830.692774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077831.012738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077831.716767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077832.036732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077832.741056:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077832.741175:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769077832.741238:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077832.741262:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077832.741307:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077832.741318:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077832.741329:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769077832.741342:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:3.0:1769077832.741414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077829/real 1769077832] req@ffff969e0a7fea00 x1855007923911680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077832.741463:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923911680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077832.741517:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077832.741524:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077832.741529:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077832.741577:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077829/real 1769077832] req@ffff969e0a7fc700 x1855007923911552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077832.741671:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923911552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077832.741681:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077832.741685:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077832.741689:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077832.741698:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077829/real 1769077832] req@ffff969e0a7fce00 x1855007923911424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077832.741713:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923911424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077832.741721:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077832.741724:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077832.741728:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077832.741736:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077829/real 1769077832] req@ffff969e0a7fe680 x1855007923911296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077832.741748:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923911296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077832.741764:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077832.741769:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077832.741773:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077832.741781:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077829/real 1769077832] req@ffff969e0a7fed80 x1855007923911168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077832.741794:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923911168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077832.741801:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077832.741805:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077832.741808:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923911168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077833.060708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077833.764738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077834.084728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077834.788736:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077834.788748:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077834.788753:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077834.788767:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4523 00000100:00080000:3.0:1769077834.788782:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077834.788791:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077834.788796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077834.788837:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923911808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077834.788873:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077834.788877:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077834.788880:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077834.788885:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4523 00000100:00080000:3.0:1769077834.788909:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077834.788915:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077834.788929:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077834.788939:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923911808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077834.788955:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923911936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077834.788973:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077834.788977:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077834.788980:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077834.788987:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4523 00000100:00080000:3.0:1769077834.789010:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077834.789017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077834.789019:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077834.789028:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077834.789051:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923912064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077834.789057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077834/real 1769077834] req@ffff969e0a7fed80 x1855007923911808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077834.789075:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077834.789090:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077834.789096:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077834.789106:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4523 00000100:00100000:2.0:1769077834.789110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923911808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077834.789116:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077834.789122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077834.789130:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769077834.789133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077834.789147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923911808:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769077834.789152:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077834.789185:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923912192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077834.789186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911936:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769077834.789233:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077834.789242:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1769077834.789244:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077834.789247:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077834.789258:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4523 00000100:00080000:3.0:1769077834.789266:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077834.789270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077834/real 1769077834] req@ffff969e0a7fe680 x1855007923911936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077834.789278:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077834.789295:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077834.789313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923911936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077834.789318:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923912320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077834.789359:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077834.789369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077834.789377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923911936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077834.789386:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077834.789413:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077834.789426:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077834/real 1769077834] req@ffff969e0a7fce00 x1855007923912064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077834.789438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923912064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077834.789459:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077834.789463:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077834.789467:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077834.789484:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077834.789529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077834.789542:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077834/real 1769077834] req@ffff969e0a7fc700 x1855007923912192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077834.789553:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923912192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077834.789562:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077834.789565:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077834.789569:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077834.789576:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077834.789639:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077834.789652:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077834/real 1769077834] req@ffff969e0a7fea00 x1855007923912320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077834.789666:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923912320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077834.789675:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077834.789679:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077834.789690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077835.108692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077835.812820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077836.132670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077836.836821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077837.156662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077837.860771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077838.180689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077838.884766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077839.204650:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077839.908722:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1769077839.908726:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077839.908737:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077839.908756:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077839.908766:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4529 00000100:00080000:0.0:1769077839.908777:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077839.908783:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077839.908812:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923912448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077839.908839:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077839.908842:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077839.908844:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077839.908847:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4529 00000100:00080000:0.0:1769077839.908849:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077839.908852:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077839.908860:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077839.908870:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923912576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077839.908877:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077839.908880:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077839.908882:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077839.908884:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4529 00000100:00080000:0.0:1769077839.908886:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077839.908889:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077839.908892:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077839.908897:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923912448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077839.908908:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923912704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077839.908929:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077839.908932:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077839.908935:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077839.908939:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4529 00000100:00080000:0.0:1769077839.908943:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077839.908946:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077839.908950:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077839.908959:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923912832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077839.908965:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077839.908968:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077839.908970:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077839.908972:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4529 00000100:00080000:0.0:1769077839.908975:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077839.908977:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077839.908980:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077839.908989:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077839.908990:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923912960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077839.909042:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077839/real 1769077839] req@ffff969e06a60a80 x1855007923912448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077839.909060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923912448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077839.909070:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077839.909074:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077839.909077:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923912448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077839.909084:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077839.909103:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077839.909112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077839/real 1769077839] req@ffff969e06a60000 x1855007923912576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077839.909120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923912576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077839.909126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077839.909128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077839.909130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077839.909153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077839.909165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077839.909185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077839/real 1769077839] req@ffff969e06a63800 x1855007923912704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077839.909194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923912704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077839.909209:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077839.909212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077839.909214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077839.909219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077839.909233:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077839.909240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077839/real 1769077839] req@ffff969e06a60700 x1855007923912832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077839.909248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923912832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077839.909254:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077839.909256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077839.909258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077839.909262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077839.909272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077839.909283:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077839/real 1769077839] req@ffff969e06a62d80 x1855007923912960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077839.909291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923912960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077839.909302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077839.909305:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077839.909307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923912960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077840.228722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077840.932758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077841.252718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077841.956708:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077842.276703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077842.980766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077843.300726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077844.004691:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077844.324705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077845.028709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077845.028720:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077845.028725:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077845.028736:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4534 00000100:00080000:2.0:1769077845.028748:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077845.028755:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077845.028793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077845.028800:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923913088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077845.028825:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077845.028829:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077845.028831:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077845.028835:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4534 00000100:00080000:2.0:1769077845.028838:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077845.028841:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077845.028850:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077845.028862:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923913216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077845.028863:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923913088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077845.028889:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077845.028891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077845.028910:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077845.028916:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4534 00000100:00080000:2.0:1769077845.028920:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077845.028926:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077845.028932:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077845.028950:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923913344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077845.028962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077845.028969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1769077845.028971:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077845.028973:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077845.028981:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4534 00000100:00080000:2.0:1769077845.028987:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077845.028991:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077845.028997:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077845.029021:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923913472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077845.029023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077845/real 1769077845] req@ffff969e0a46e300 x1855007923913088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077845.029077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077845.029083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923913088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077845.029084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077845.029090:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077845.029102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077845.029109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769077845.029114:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4534 00000100:00080000:2.0:1769077845.029120:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077845.029133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923913088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077845.029135:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077845.029146:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077845.029169:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077845.029170:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923913600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077845.029209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077845.029231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077845/real 1769077845] req@ffff969e0a46ed80 x1855007923913216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077845.029240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923913216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077845.029246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077845.029249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077845.029252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077845.029257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077845.029273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077845.029284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077845/real 1769077845] req@ffff969e0a46f100 x1855007923913344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077845.029296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923913344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077845.029305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077845.029309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077845.029312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077845.029319:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077845.029366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077845.029382:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077845/real 1769077845] req@ffff969e0a46df80 x1855007923913472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077845.029391:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923913472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077845.029397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077845.029399:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077845.029402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077845.029416:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077845.029429:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077845.029436:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077845/real 1769077845] req@ffff969e0a46ca80 x1855007923913600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077845.029443:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923913600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077845.029449:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077845.029451:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077845.029453:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077845.348711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077846.052815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077846.372744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077847.076830:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077847.396729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077848.100853:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077848.420721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077849.124797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077849.444736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077850.148729:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077850.148739:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077850.148743:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077850.148770:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4539 00000100:00080000:3.0:1769077850.148781:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077850.148787:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077850.148795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077850.148816:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923913728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077850.148840:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077850.148843:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077850.148844:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077850.148847:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4539 00000100:00080000:3.0:1769077850.148850:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077850.148855:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077850.148864:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077850.148873:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923913856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077850.148883:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077850.148885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077850.148887:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077850.148895:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4539 00000100:00080000:3.0:1769077850.148898:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077850.148901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923913728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077850.148902:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077850.148913:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077850.148922:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923913984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077850.148942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077850.148946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077850.148948:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077850.148952:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4539 00000100:00080000:3.0:1769077850.148959:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077850.148964:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077850.148970:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077850.148985:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923914112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077850.148995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077850.149002:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769077850.149005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077850.149009:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077850.149015:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4539 00000100:00080000:3.0:1769077850.149022:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077850.149027:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077850.149033:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077850.149046:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923914240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077850.149050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077850/real 1769077850] req@ffff969e0a7fdf80 x1855007923913728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077850.149110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923913728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077850.149120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077850.149125:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077850.149128:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923913728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077850.149171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077850.149190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077850.149198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077850/real 1769077850] req@ffff969e0a7fca80 x1855007923913856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077850.149206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923913856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077850.149213:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077850.149215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077850.149218:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077850.149222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077850.149279:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077850.149287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077850/real 1769077850] req@ffff969e0a7fe300 x1855007923913984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077850.149295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923913984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077850.149301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077850.149303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077850.149305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923913984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077850.149310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077850.149323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077850.149329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077850/real 1769077850] req@ffff969e0a7fd180 x1855007923914112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077850.149336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923914112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077850.149342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077850.149343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077850.149345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077850.149350:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077850.149359:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077850.149365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077850/real 1769077850] req@ffff969e0a7fd880 x1855007923914240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077850.149383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923914240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077850.149393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077850.149397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077850.149400:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077850.468671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077851.172749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077851.492719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077852.196746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077852.516690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077853.220807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077853.540714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077854.244864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077854.564662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769077855.268661:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077855.268670:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077855.268674:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077855.268683:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4544 00000100:00080000:3.0:1769077855.268710:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077855.268716:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769077855.268723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077855.268749:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923914368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077855.268779:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077855.268783:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077855.268785:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077855.268790:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4544 00000100:00080000:3.0:1769077855.268794:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077855.268799:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077855.268808:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077855.268820:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923914496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077855.268827:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077855.268830:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077855.268831:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077855.268835:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4544 00000100:00080000:3.0:1769077855.268838:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077855.268840:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077855.268843:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077855.268852:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923914624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077855.268859:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077855.268867:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769077855.268869:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077855.268872:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4544 00000100:00080000:3.0:1769077855.268875:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077855.268878:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077855.268883:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077855.268894:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923914752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077855.268896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923914368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769077855.268920:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077855.268930:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769077855.268933:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077855.268937:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4544 00000100:00080000:3.0:1769077855.268941:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077855.268944:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769077855.268947:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077855.268955:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923914880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077855.269013:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077855.269062:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077855/real 1769077855] req@ffff969e0a7fd500 x1855007923914368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077855.269078:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923914368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077855.269087:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077855.269091:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077855.269094:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923914368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077855.269100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077855.269116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077855.269124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077855/real 1769077855] req@ffff969e0a7fdc00 x1855007923914496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077855.269132:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923914496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077855.269159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077855.269176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077855.269180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077855.269187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077855.269217:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077855.269229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077855/real 1769077855] req@ffff969e0a7ff800 x1855007923914624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077855.269242:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923914624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077855.269251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077855.269255:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077855.269258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077855.269289:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077855.269306:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077855.269314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077855/real 1769077855] req@ffff969e0a7fc000 x1855007923914752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077855.269321:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923914752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077855.269327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077855.269329:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077855.269337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077855.269341:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077855.269355:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077855.269361:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077855/real 1769077855] req@ffff969e0a7fc380 x1855007923914880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077855.269369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923914880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077855.269375:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077855.269377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077855.269379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923914880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077855.588747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077856.292762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077856.612677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077857.317364:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077857.636714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077858.341118:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077858.660701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077859.365325:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077859.684690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077860.388722:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077860.388738:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077860.388743:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077860.388756:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4549 00000100:00080000:2.0:1769077860.388787:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077860.388796:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077860.388840:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923915008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077860.388893:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077860.388897:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077860.388899:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077860.388905:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4549 00000100:00080000:2.0:1769077860.388910:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077860.388928:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077860.388940:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077860.388956:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923915136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077860.388966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077860.388970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077860.388972:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077860.388976:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4549 00000100:00080000:2.0:1769077860.388981:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077860.388985:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077860.388991:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077860.389002:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923915264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077860.389014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077860.389018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077860.389020:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077860.389025:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4549 00000100:00080000:2.0:1769077860.389029:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077860.389033:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077860.389038:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077860.389049:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923915392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077860.389059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077860.389069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077860.389111:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077860.389116:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4549 00000100:00080000:2.0:1769077860.389120:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077860.389124:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077860.389130:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077860.389144:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923915520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769077860.389638:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077860.390002:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923915008:192.168.204.143@tcp:250:kworker.0 00000800:00000100:3.0:1769077860.390093:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077860.390131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077860/real 1769077860] req@ffff969e0a46d500 x1855007923915008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077860.390164:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923915008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077860.390176:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077860.390181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077860.390186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923915008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077860.390195:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077860.390224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077860.390255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077860/real 1769077860] req@ffff969e0a46ea00 x1855007923915136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077860.390269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923915136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077860.390279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077860.390282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077860.390286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077860.390293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077860.390310:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077860.390318:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077860/real 1769077860] req@ffff969e0a46f480 x1855007923915264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077860.390326:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923915264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077860.390342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077860.390344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077860.390347:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077860.390352:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077860.390366:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077860.390373:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077860/real 1769077860] req@ffff969e0a46f800 x1855007923915392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077860.390380:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923915392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077860.390386:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077860.390388:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077860.390390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077860.390394:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077860.390404:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077860.390410:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077860/real 1769077860] req@ffff969e0a46ce00 x1855007923915520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077860.390417:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923915520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077860.390422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077860.390433:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077860.390443:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077860.708679:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077861.412835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077861.732652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077862.436984:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077862.756721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077863.460834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077863.780722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077864.484786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077864.804735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077865.508726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077865.508738:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077865.508743:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077865.508754:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4554 00000100:00080000:2.0:1769077865.508767:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077865.508775:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077865.508791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077865.508820:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923915648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077865.508848:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077865.508852:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077865.508854:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077865.508859:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4554 00000100:00080000:2.0:1769077865.508863:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077865.508868:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077865.508880:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077865.508893:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923915776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077865.508903:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923915648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077865.508932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077865.508935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077865.508937:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077865.508941:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4554 00000100:00080000:2.0:1769077865.508945:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077865.508948:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077865.508953:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077865.508964:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923915904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077865.508971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077865.508974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077865.508975:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077865.508979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077865.508983:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4554 00000100:00080000:2.0:1769077865.508999:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077865.509005:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077865.509011:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077865.509027:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923916032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077865.509035:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077865/real 1769077865] req@ffff969e0a46c380 x1855007923915648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077865.509041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077865.509049:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077865.509051:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077865.509056:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4554 00000100:00100000:3.0:1769077865.509058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923915648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077865.509068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077865.509071:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077865.509072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769077865.509080:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077865.509086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923915648:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769077865.509089:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077865.509105:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923916160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077865.509112:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077865.509134:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077865.509145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077865/real 1769077865] req@ffff969e0a46d880 x1855007923915776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077865.509153:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923915776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077865.509159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077865.509161:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077865.509163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077865.509174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077865.509188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077865.509194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077865/real 1769077865] req@ffff969e0a46ce00 x1855007923915904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077865.509202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923915904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077865.509207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077865.509209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077865.509211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923915904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077865.509214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077865.509225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077865.509230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077865/real 1769077865] req@ffff969e0a46f800 x1855007923916032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077865.509237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923916032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077865.509242:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077865.509244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077865.509246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077865.509259:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077865.509268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077865.509274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077865/real 1769077865] req@ffff969e0a46f480 x1855007923916160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077865.509287:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923916160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077865.509294:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077865.509296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077865.509298:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077865.828692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077866.532798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077866.852744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077867.556871:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077867.876675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077868.580784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077868.900752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077869.604748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077869.924729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077870.628735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077870.628744:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077870.628749:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077870.628758:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4559 00000100:00080000:2.0:1769077870.628769:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077870.628776:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077870.628810:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923916288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769077870.628833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077870.628843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077870.628847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077870.628849:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077870.628855:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4559 00000100:00080000:2.0:1769077870.628860:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077870.628863:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077870.628871:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077870.628883:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923916416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077870.628891:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077870.628927:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077870.628929:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077870.628933:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4559 00000100:00080000:2.0:1769077870.628937:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077870.628940:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077870.628945:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077870.628951:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923916288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077870.628959:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923916544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077870.628974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077870.628976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077870.628977:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077870.628981:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4559 00000100:00080000:2.0:1769077870.628984:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077870.628987:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077870.628990:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077870.628999:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923916672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077870.629011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077870.629014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077870.629015:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077870.629018:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4559 00000100:00080000:2.0:1769077870.629022:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077870.629026:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077870.629038:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077870.629054:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923916800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077870.629064:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077870.629122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077870/real 1769077870] req@ffff969e0a46ea00 x1855007923916288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077870.629146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923916288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077870.629158:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077870.629164:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077870.629169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923916288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077870.629180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077870.629210:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077870.629223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077870/real 1769077870] req@ffff969e0a46d500 x1855007923916416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077870.629236:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923916416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077870.629245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077870.629250:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077870.629254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077870.629308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077870.629335:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077870.629348:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077870/real 1769077870] req@ffff969e0a46ca80 x1855007923916544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077870.629362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923916544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077870.629372:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077870.629376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077870.629380:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077870.629388:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077870.629408:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077870.629418:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077870/real 1769077870] req@ffff969e0a46df80 x1855007923916672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077870.629431:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923916672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077870.629439:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077870.629443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077870.629446:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077870.629453:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077870.629475:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077870.629487:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077870/real 1769077870] req@ffff969e0a46f100 x1855007923916800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077870.629510:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923916800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077870.629520:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077870.629524:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077870.629528:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923916800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077870.948820:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077871.652828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077871.972685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077872.676831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077872.996720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077873.700801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077874.020691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077874.724757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077875.044779:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077875.748727:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077875.748740:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077875.748745:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077875.748756:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4564 00000100:00080000:0.0:1769077875.748782:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077875.748791:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077875.748797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077875.748870:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923916928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077875.748898:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077875.748902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077875.748905:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077875.748909:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4564 00000100:00080000:0.0:1769077875.748914:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077875.748919:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077875.748930:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077875.748934:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923916928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077875.748942:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923917056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077875.748976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077875.748979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077875.748981:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077875.748985:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4564 00000100:00080000:0.0:1769077875.748988:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077875.748991:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077875.748995:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077875.749005:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923917184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077875.749009:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077875.749014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077875.749017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077875.749019:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077875.749023:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4564 00000100:00080000:0.0:1769077875.749026:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077875.749029:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077875.749033:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077875.749041:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923917312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077875.749048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077875/real 1769077875] req@ffff969e06a61180 x1855007923916928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077875.749050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077875.749053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077875.749055:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077875.749058:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4564 00000100:00100000:3.0:1769077875.749067:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923916928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077875.749079:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077875.749079:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077875.749084:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:3.0:1769077875.749084:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077875.749089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923916928:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077875.749090:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077875.749100:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923917440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077875.749121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077875.749146:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077875.749160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077875/real 1769077875] req@ffff969e06a62680 x1855007923917056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077875.749171:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923917056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077875.749177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077875.749179:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077875.749181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077875.749186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077875.749200:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077875.749206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077875/real 1769077875] req@ffff969e06a61f80 x1855007923917184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077875.749213:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923917184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077875.749218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077875.749220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077875.749221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077875.749225:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077875.749236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077875.749242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077875/real 1769077875] req@ffff969e06a60380 x1855007923917312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077875.749257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923917312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077875.749263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077875.749265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077875.749267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077875.749271:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077875.749283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077875.749289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077875/real 1769077875] req@ffff969e06a61500 x1855007923917440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077875.749296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923917440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077875.749301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077875.749303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077875.749305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077876.068686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077876.772774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077877.092718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077877.796856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077878.116661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077878.820717:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077879.140736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077879.844954:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077880.164716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077880.868727:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077880.868751:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077880.868756:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769077880.868760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077880.868768:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4570 00000100:00080000:0.0:1769077880.868781:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077880.868787:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077880.868825:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923917568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077880.868845:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077880.868848:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077880.868850:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077880.868854:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4570 00000100:00080000:0.0:1769077880.868858:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077880.868861:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077880.868871:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077880.868883:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923917696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077880.868883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923917568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077880.868900:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077880.868921:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077880.868923:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077880.868926:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4570 00000100:00080000:0.0:1769077880.868930:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077880.868933:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077880.868937:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077880.868949:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923917824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077880.868949:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077880.868957:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077880.868960:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077880.868962:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077880.868972:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4570 00000100:00080000:0.0:1769077880.868976:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077880.868979:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077880.868982:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077880/real 1769077880] req@ffff969e06a61880 x1855007923917568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077880.868983:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077880.868993:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923917952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077880.869002:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077880.869005:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077880.869007:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077880.869010:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4570 00000100:00100000:3.0:1769077880.869012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923917568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077880.869014:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077880.869018:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:3.0:1769077880.869024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769077880.869025:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769077880.869028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077880.869032:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923917568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077880.869041:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923918080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077880.869056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077880.869077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077880.869089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077880/real 1769077880] req@ffff969e06a63100 x1855007923917696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077880.869100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923917696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077880.869109:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077880.869112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077880.869115:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077880.869121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077880.869140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077880.869148:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077880/real 1769077880] req@ffff969e06a61500 x1855007923917824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077880.869156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923917824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077880.869169:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077880.869172:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077880.869174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077880.869178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077880.869190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077880.869196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077880/real 1769077880] req@ffff969e06a60380 x1855007923917952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077880.869203:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923917952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077880.869208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077880.869211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077880.869214:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923917952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077880.869219:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077880.869236:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077880.869246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077880/real 1769077880] req@ffff969e06a61f80 x1855007923918080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077880.869257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923918080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077880.869265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077880.869268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077880.869277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077881.188653:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077881.892813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077882.212718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077882.916829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077883.236721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077883.940772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077884.260725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077884.964819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077885.284738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077885.988966:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077885.988987:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077885.988992:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077885.989005:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4575 00000100:00080000:0.0:1769077885.989017:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077885.989025:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077885.989070:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923918208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077885.989100:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077885.989104:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077885.989106:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077885.989113:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4575 00000100:00080000:0.0:1769077885.989118:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077885.989124:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077885.989135:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077885.989151:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923918336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769077885.989156:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077885.989162:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077885.989166:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077885.989168:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077885.989172:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4575 00000100:00080000:0.0:1769077885.989193:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077885.989199:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077885.989205:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077885.989219:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923918464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077885.989230:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077885.989233:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077885.989235:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077885.989239:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4575 00000100:00080000:0.0:1769077885.989243:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077885.989246:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077885.989251:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077885.989262:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923918592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077885.989270:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077885.989273:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077885.989276:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077885.989279:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4575 00000100:00080000:0.0:1769077885.989282:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077885.989286:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077885.989290:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077885.989300:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923918720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077885.989984:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923918208:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769077885.990053:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077885.990078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077885/real 1769077885] req@ffff969e06a62680 x1855007923918208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077885.990094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923918208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077885.990102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077885.990106:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077885.990109:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923918208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077885.990114:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077885.990136:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077885.990144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077885/real 1769077885] req@ffff969e06a61180 x1855007923918336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077885.990151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923918336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077885.990166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077885.990169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077885.990171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077885.990176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077885.990189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077885.990194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077885/real 1769077885] req@ffff969e06a62d80 x1855007923918464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077885.990201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923918464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077885.990207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077885.990209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077885.990211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077885.990214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077885.990224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077885.990229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077885/real 1769077885] req@ffff969e06a60700 x1855007923918592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077885.990238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923918592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077885.990246:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077885.990249:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077885.990258:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077885.990265:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077885.990289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077885.990299:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077885/real 1769077885] req@ffff969e06a63800 x1855007923918720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077885.990307:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923918720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077885.990313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077885.990315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077885.990317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077886.308783:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077887.012770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077887.332720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077888.036756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077888.356711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077889.060755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077889.380702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077890.084694:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077890.404718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077891.108729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077891.108747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077891.108759:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077891.108764:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077891.108777:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4580 00000100:00080000:0.0:1769077891.108788:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077891.108796:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077891.108844:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923918848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077891.108877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077891.108882:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077891.108884:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077891.108890:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4580 00000100:00080000:0.0:1769077891.108894:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077891.108899:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077891.108911:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077891.108935:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923918976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077891.108936:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923918848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077891.108948:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077891.108951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077891.108954:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077891.108961:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4580 00000100:00080000:0.0:1769077891.108965:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077891.108968:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077891.108988:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077891.109001:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923919104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077891.109013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077891.109017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077891.109019:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077891.109023:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4580 00000100:00080000:0.0:1769077891.109028:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077891.109032:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077891.109038:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077891.109057:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077891.109059:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923919232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077891.109071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077891.109074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077891.109076:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077891.109079:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4580 00000100:00080000:0.0:1769077891.109084:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077891.109087:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077891.109092:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077891.109100:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923919360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077891.109122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077891/real 1769077891] req@ffff969e06a60000 x1855007923918848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077891.109143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923918848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077891.109155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077891.109160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077891.109163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923918848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077891.109186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077891.109206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077891.109223:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077891/real 1769077891] req@ffff969e06a60a80 x1855007923918976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077891.109232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923918976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077891.109238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077891.109240:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077891.109242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923918976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077891.109255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077891.109277:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077891.109289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077891/real 1769077891] req@ffff969e06a63800 x1855007923919104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077891.109302:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923919104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077891.109311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077891.109315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077891.109318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077891.109325:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077891.109342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077891.109349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077891/real 1769077891] req@ffff969e06a60700 x1855007923919232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077891.109357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923919232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077891.109369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077891.109371:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077891.109373:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077891.109377:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077891.109391:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077891.109397:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077891/real 1769077891] req@ffff969e06a62d80 x1855007923919360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077891.109405:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923919360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077891.109410:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077891.109412:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077891.109414:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077891.428723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077892.132737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077892.452701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077893.156731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077893.476688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077894.180735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077894.500719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077895.204731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077895.524652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077896.228707:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077896.228715:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077896.228724:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077896.228746:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077896.228759:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4585 00000100:00080000:0.0:1769077896.228772:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077896.228779:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077896.228813:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923919488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077896.228839:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077896.228842:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077896.228844:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077896.228849:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4585 00000100:00080000:0.0:1769077896.228853:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077896.228857:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077896.228867:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077896.228878:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923919488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077896.228884:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923919616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077896.228897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077896.228901:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077896.228903:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077896.228907:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4585 00000100:00080000:0.0:1769077896.228911:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077896.228916:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077896.228921:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077896.228934:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923919744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077896.228956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077896.228960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077896.228963:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077896.228968:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4585 00000100:00080000:0.0:1769077896.228973:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077896.228979:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077896.228985:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077896.228997:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077896.229001:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923919872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077896.229015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077896.229019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077896.229021:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077896.229025:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4585 00000100:00080000:0.0:1769077896.229031:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077896.229034:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919744:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769077896.229040:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077896.229047:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077896.229063:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923920000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077896.229070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077896.229098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077896.548697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077897.252728:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077897.572717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077898.276699:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077898.596711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077899.300733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1769077899.300943:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:3.0:1769077899.300996:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769077899.301019:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077899.301072:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077899.301083:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077899.301092:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077899.301102:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769077899.301151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077896/real 1769077899] req@ffff969e06a61500 x1855007923920000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.301209:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923920000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.301225:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.301231:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077899.301237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077899.301246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077896/real 1769077899] req@ffff969e06a60380 x1855007923919872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.301260:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923919872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.301268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.301273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077899.301277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077899.301285:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077896/real 1769077899] req@ffff969e06a62680 x1855007923919616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.301299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923919616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.301308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.301312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077899.301317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077899.301327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077896/real 1769077899] req@ffff969e06a61f80 x1855007923919744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.301370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923919744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.301380:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.301386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077899.301390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923919744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077899.301398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077896/real 1769077899] req@ffff969e06a61180 x1855007923919488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077899.301428:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923919488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077899.301435:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077899.301438:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077899.301442:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923919488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769077899.620669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077900.324758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077900.644703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077901.348701:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077901.348710:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077901.348715:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077901.348723:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4590 00010000:00010000:1.0:1769077901.348734:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077901.348752:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077901.348759:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077901.348791:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923920128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077901.348823:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077901.348827:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077901.348829:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077901.348834:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4590 00000100:00080000:0.0:1769077901.348838:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077901.348842:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077901.348853:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077901.348866:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923920256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077901.348876:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077901.348879:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077901.348881:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077901.348885:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4590 00000100:00080000:0.0:1769077901.348888:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077901.348891:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923920128:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077901.348892:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077901.348897:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077901.348907:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923920384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077901.348916:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077901.348925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077901.348927:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077901.348930:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4590 00000100:00080000:0.0:1769077901.348934:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077901.348937:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077901.348942:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077901.348959:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077901.348960:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923920512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077901.348968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077901.348971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077901.348973:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077901.348976:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4590 00000100:00080000:0.0:1769077901.348979:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077901.348982:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077901.348986:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077901.348993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077901/real 1769077901] req@ffff969e06a63100 x1855007923920128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077901.348995:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923920640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077901.349014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923920128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077901.349026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077901.349031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077901.349035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923920128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077901.349057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077901.349083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077901.349096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077901/real 1769077901] req@ffff969e06a61880 x1855007923920256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077901.349108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923920256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077901.349118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077901.349122:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077901.349126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077901.349155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077901.349175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077901.349187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077901/real 1769077901] req@ffff969e06a61180 x1855007923920384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077901.349200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923920384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077901.349222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077901.349226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077901.349229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077901.349235:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077901.349257:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077901.349269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077901/real 1769077901] req@ffff969e06a61f80 x1855007923920512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077901.349292:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923920512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077901.349301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077901.349304:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077901.349307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077901.349312:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077901.349334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077901.349345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077901/real 1769077901] req@ffff969e06a62680 x1855007923920640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077901.349357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923920640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077901.349364:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077901.349367:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077901.349371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077901.668700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077902.372747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077902.692711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077903.396756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077903.716710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077904.420726:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077904.740725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077905.444720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077905.764736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077906.468698:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077906.468707:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1769077906.468710:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077906.468712:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077906.468721:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4595 00000100:00080000:0.0:1769077906.468732:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077906.468738:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077906.468772:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923920768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077906.468812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077906.468815:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077906.468816:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077906.468819:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4595 00000100:00080000:0.0:1769077906.468822:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077906.468825:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077906.468833:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077906.468844:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923920896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077906.468851:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077906.468854:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077906.468856:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077906.468858:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4595 00000100:00080000:0.0:1769077906.468861:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077906.468864:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923920768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077906.468865:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077906.468871:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077906.468879:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923921024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077906.468886:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077906.468888:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077906.468889:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077906.468892:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4595 00000100:00080000:0.0:1769077906.468900:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077906.468903:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077906.468907:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077906.468915:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923921152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077906.468922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077906.468925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077906.468926:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077906.468931:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4595 00000800:00000100:2.0:1769077906.468932:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077906.468934:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077906.468938:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077906.468941:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077906.468950:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923921280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077906.468969:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077906/real 1769077906] req@ffff969e06a60380 x1855007923920768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077906.468983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923920768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077906.469002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077906.469005:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077906.469009:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923920768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077906.469014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077906.469032:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077906.469040:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077906/real 1769077906] req@ffff969e06a61500 x1855007923920896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077906.469049:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923920896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077906.469054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077906.469057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077906.469059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923920896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077906.469085:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077906.469098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077906.469104:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077906/real 1769077906] req@ffff969e06a62d80 x1855007923921024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077906.469112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923921024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077906.469118:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077906.469120:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077906.469122:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077906.469132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077906.469145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077906.469151:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077906/real 1769077906] req@ffff969e06a60700 x1855007923921152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077906.469165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923921152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077906.469172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077906.469174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077906.469176:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077906.469180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077906.469191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077906.469197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077906/real 1769077906] req@ffff969e06a63800 x1855007923921280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077906.469204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923921280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077906.469210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077906.469212:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077906.469213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077906.788697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077907.492755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077907.812711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077908.516727:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077908.836727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077909.540717:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077909.860704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077910.564754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077910.884692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077911.588691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077911.588702:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077911.588706:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077911.588717:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4600 00000100:00080000:0.0:1769077911.588729:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1769077911.588731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077911.588736:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077911.588774:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923921408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077911.588798:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077911.588817:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077911.588819:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077911.588823:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4600 00000100:00080000:0.0:1769077911.588827:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077911.588831:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077911.588840:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077911.588841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923921408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077911.588852:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923921536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077911.588861:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077911.588864:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077911.588866:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077911.588869:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4600 00000100:00080000:0.0:1769077911.588871:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077911.588874:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077911.588877:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077911.588887:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923921664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077911.588893:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077911.588896:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077911.588904:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077911.588907:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4600 00000100:00080000:0.0:1769077911.588910:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769077911.588912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077911.588918:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077911.588929:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077911.588938:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923921792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077911.588947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077911.588947:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077911/real 1769077911] req@ffff969e06a60a80 x1855007923921408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077911.588950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077911.588952:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077911.588956:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4600 00000100:00080000:0.0:1769077911.588963:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077911.588968:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077911.588972:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077911.588984:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923921920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077911.588984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923921408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077911.589014:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077911.589020:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077911.589026:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923921408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077911.589052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077911.589070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077911.589080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077911/real 1769077911] req@ffff969e06a60000 x1855007923921536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077911.589089:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923921536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077911.589095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077911.589097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077911.589099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077911.589103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077911.589113:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077911.589119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077911/real 1769077911] req@ffff969e06a63800 x1855007923921664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077911.589127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923921664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077911.589141:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077911.589144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077911.589146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077911.589150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077911.589164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077911.589170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077911/real 1769077911] req@ffff969e06a60700 x1855007923921792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077911.589178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923921792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077911.589184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077911.589186:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077911.589188:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077911.589191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077911.589201:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077911.589206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077911/real 1769077911] req@ffff969e06a62d80 x1855007923921920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077911.589226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923921920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077911.589233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077911.589235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077911.589245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923921920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077911.908707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077912.612725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077912.932666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077913.636722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077913.956708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077914.660726:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077914.980684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077915.684708:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077916.004702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077916.708698:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077916.708710:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077916.708716:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769077916.708722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077916.708727:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4605 00000100:00080000:0.0:1769077916.708745:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077916.708751:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077916.708784:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923922048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077916.708811:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077916.708814:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077916.708816:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077916.708819:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4605 00000100:00080000:0.0:1769077916.708836:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077916.708840:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077916.708848:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077916.708861:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923922176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077916.708862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923922048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077916.708877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077916.708880:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077916.708881:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077916.708885:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4605 00000100:00080000:0.0:1769077916.708888:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077916.708893:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077916.708898:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077916.708911:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923922304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077916.708921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077916.708925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077916.708927:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769077916.708932:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077916.708936:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4605 00000100:00080000:0.0:1769077916.708944:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077916.708949:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077916.708954:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077916.708965:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923922432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077916.708970:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077916/real 1769077916] req@ffff969e06a61500 x1855007923922048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077916.708989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769077916.708990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923922048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077916.708994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077916.708998:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077916.709007:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4605 00000100:00080000:2.0:1769077916.709010:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077916.709012:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077916.709018:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769077916.709019:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077916.709026:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077916.709027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923922048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077916.709037:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923922560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077916.709053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077916.709069:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077916.709077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077916/real 1769077916] req@ffff969e06a60380 x1855007923922176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077916.709086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923922176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077916.709092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077916.709094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077916.709096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077916.709100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077916.709110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077916.709125:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077916/real 1769077916] req@ffff969e06a62680 x1855007923922304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077916.709159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923922304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077916.709165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077916.709167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077916.709170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077916.709174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077916.709190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077916.709196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077916/real 1769077916] req@ffff969e06a61f80 x1855007923922432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077916.709204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923922432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077916.709210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077916.709211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077916.709213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077916.709217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077916.709226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077916.709232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077916/real 1769077916] req@ffff969e06a61180 x1855007923922560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077916.709239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923922560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077916.709250:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077916.709253:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077916.709255:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077917.028702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077917.732714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077918.052697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077918.756727:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077919.076724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077919.780695:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077920.100712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077920.804762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077921.124706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077921.828714:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077921.828725:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077921.828728:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077921.828737:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4611 00010000:00010000:1.0:1769077921.828748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077921.828749:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077921.828756:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077921.828791:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923922688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077921.828817:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077921.828820:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077921.828822:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077921.828825:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4611 00000100:00080000:0.0:1769077921.828828:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077921.828830:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077921.828853:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077921.828864:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923922816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077921.828875:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077921.828879:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077921.828881:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077921.828886:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4611 00000100:00100000:2.0:1769077921.828892:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923922688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077921.828894:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077921.828904:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077921.828910:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077921.828926:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923922944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077921.828934:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077921.828937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077921.828939:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077921.828942:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4611 00000100:00080000:0.0:1769077921.828945:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077921.828948:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077921.828951:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077921.828960:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923923072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077921.828966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077921.828971:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077921.828972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077921.828982:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077921.828988:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4611 00000100:00080000:0.0:1769077921.828994:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077921.828998:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077921.829004:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077921.829016:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077921/real 1769077921] req@ffff969e06a61880 x1855007923922688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077921.829021:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923923200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077921.829061:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923922688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077921.829075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077921.829080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077921.829085:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923922688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077921.829093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077921.829117:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077921.829130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077921/real 1769077921] req@ffff969e06a63100 x1855007923922816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077921.829144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923922816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077921.829153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077921.829157:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077921.829161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077921.829194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077921.829216:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077921.829239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077921/real 1769077921] req@ffff969e06a61180 x1855007923922944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077921.829257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923922944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077921.829266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077921.829270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077921.829274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923922944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077921.829281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077921.829303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077921.829315:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077921/real 1769077921] req@ffff969e06a61f80 x1855007923923072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077921.829328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923923072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077921.829337:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077921.829341:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077921.829345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077921.829351:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077921.829371:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077921.829381:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077921/real 1769077921] req@ffff969e06a62680 x1855007923923200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077921.829404:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923923200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077921.829413:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077921.829416:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077921.829419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077922.148716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077922.852760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077923.172711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077923.877848:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077924.196695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077924.900803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077925.220704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077925.925174:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077926.244710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077926.949023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077926.949038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077926.949044:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077926.949092:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4616 00000100:00080000:1.0:1769077926.949107:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077926.949117:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077926.949182:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923923328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077926.949222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077926.949226:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077926.949228:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077926.949234:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4616 00000100:00080000:1.0:1769077926.949238:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077926.949261:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077926.949272:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077926.949291:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923923456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077926.949302:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077926.949306:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077926.949308:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077926.949312:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4616 00000100:00080000:1.0:1769077926.949317:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077926.949321:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077926.949326:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077926.949339:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923923584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077926.949351:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077926.949355:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769077926.949357:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077926.949363:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4616 00000100:00080000:1.0:1769077926.949369:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077926.949374:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077926.949380:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077926.949393:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923923712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077926.949403:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077926.949413:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077926.949415:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077926.949422:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4616 00000100:00080000:1.0:1769077926.949428:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077926.949432:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077926.949437:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077926.949453:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923923840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1769077926.949516:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077926.949630:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923923328:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769077926.949720:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077926.949758:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077926/real 1769077926] req@ffff969e10b60700 x1855007923923328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077926.949773:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923923328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077926.949781:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077926.949784:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077926.949787:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923923328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077926.949794:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077926.949811:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077926.949820:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077926/real 1769077926] req@ffff969e10b62d80 x1855007923923456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077926.949828:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923923456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077926.949844:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077926.949847:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077926.949849:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077926.949853:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077926.949867:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077926.949873:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077926/real 1769077926] req@ffff969e10b60000 x1855007923923584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077926.949881:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923923584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077926.949886:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077926.949888:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077926.949889:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077926.949893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077926.949904:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077926.949909:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077926/real 1769077926] req@ffff969e10b61180 x1855007923923712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077926.949917:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923923712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077926.949922:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077926.949923:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077926.949930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077926.949935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077926.949969:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077926.949982:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077926/real 1769077926] req@ffff969e10b60380 x1855007923923840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077926.949995:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923923840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077926.950004:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077926.950008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077926.950012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923923840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077927.268736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077927.973246:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077928.292705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077928.996790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077929.316761:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077930.020793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077930.340730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077931.044769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077931.364751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077932.068744:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077932.068757:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077932.068762:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769077932.068771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077932.068771:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4621 00000100:00080000:2.0:1769077932.068784:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077932.068791:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077932.068825:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923923968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077932.068852:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077932.068855:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077932.068856:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077932.068859:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4621 00000100:00080000:2.0:1769077932.068861:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077932.068898:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077932.068909:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077932.068918:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923923968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077932.068927:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923924096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077932.068941:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077932.068946:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077932.068948:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077932.068953:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4621 00000100:00080000:2.0:1769077932.068958:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077932.068963:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077932.068967:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077932.068978:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923924224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077932.068986:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077932.068992:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1769077932.068992:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077932.068993:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077932.068999:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4621 00000100:00080000:2.0:1769077932.069002:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077932.069005:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077932.069009:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077932.069016:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923924352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077932.069023:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077932.069030:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077932.069033:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077932.069034:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077932/real 1769077932] req@ffff969e0a46ed80 x1855007923923968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077932.069040:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4621 00000100:00080000:2.0:1769077932.069047:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077932.069052:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077932.069056:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077932.069060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923923968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077932.069065:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923924480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077932.069081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077932.069087:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077932.069091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923923968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077932.069113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077932.069130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077932.069139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077932/real 1769077932] req@ffff969e0a46e300 x1855007923924096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077932.069148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923924096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077932.069164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077932.069167:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077932.069169:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077932.069174:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077932.069188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077932.069195:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077932/real 1769077932] req@ffff969e0a46f100 x1855007923924224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077932.069202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923924224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077932.069208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077932.069209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077932.069211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077932.069215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077932.069225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077932.069231:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077932/real 1769077932] req@ffff969e0a46df80 x1855007923924352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077932.069239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923924352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077932.069244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077932.069246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077932.069252:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077932.069257:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077932.069270:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077932.069276:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077932/real 1769077932] req@ffff969e0a46ca80 x1855007923924480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077932.069284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923924480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077932.069290:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077932.069291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077932.069293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077932.388661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077933.092725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077933.412707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077934.116793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077934.436698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077935.140729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077935.460700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077936.164704:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077936.484718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077937.188687:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077937.188696:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077937.188707:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077937.188713:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077937.188722:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4626 00000100:00080000:0.0:1769077937.188733:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077937.188780:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077937.188839:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923924608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077937.188862:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077937.188865:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077937.188867:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077937.188870:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4626 00000100:00080000:0.0:1769077937.188873:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077937.188877:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077937.188884:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077937.188893:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923924736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077937.188918:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077937.188923:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769077937.188924:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923924608:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077937.188926:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077937.188936:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4626 00000100:00080000:0.0:1769077937.188941:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077937.188946:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077937.188952:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077937.188977:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923924864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077937.188987:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077937.188990:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077937.188992:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077937.188996:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4626 00000100:00080000:0.0:1769077937.188999:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769077937.189001:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077937.189006:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077937.189013:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077937.189024:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923924992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077937.189034:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077937.189038:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077937.189040:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077937.189047:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4626 00000100:00100000:3.0:1769077937.189047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077937/real 1769077937] req@ffff969e06a60380 x1855007923924608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077937.189061:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077937.189068:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077937.189074:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077937.189079:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923924608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077937.189089:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923925120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077937.189097:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077937.189105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077937.189116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923924608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077937.189143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077937.189168:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077937.189187:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077937/real 1769077937] req@ffff969e06a61500 x1855007923924736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077937.189197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923924736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077937.189203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077937.189205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077937.189208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077937.189212:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077937.189225:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077937.189232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077937/real 1769077937] req@ffff969e06a62d80 x1855007923924864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077937.189240:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923924864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077937.189245:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077937.189246:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077937.189248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077937.189252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077937.189263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077937.189269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077937/real 1769077937] req@ffff969e06a60700 x1855007923924992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077937.189276:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923924992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077937.189287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077937.189291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077937.189294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923924992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077937.189300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077937.189321:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077937.189333:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077937/real 1769077937] req@ffff969e06a63800 x1855007923925120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077937.189346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923925120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077937.189355:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077937.189358:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077937.189361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077937.508740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077938.212723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077938.532724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077939.236733:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077939.556694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077940.260747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077940.580767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077941.284753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077941.604734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077942.308721:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077942.308731:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077942.308735:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077942.308744:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4631 00000100:00080000:0.0:1769077942.308754:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1769077942.308757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077942.308760:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077942.308798:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923925248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077942.308823:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077942.308826:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077942.308827:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077942.308831:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4631 00000100:00080000:0.0:1769077942.308833:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077942.308836:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077942.308843:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077942.308852:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923925376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077942.308860:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077942.308876:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077942.308878:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077942.308881:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4631 00000100:00080000:0.0:1769077942.308884:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077942.308889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923925248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077942.308890:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077942.308897:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077942.308908:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923925504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077942.308916:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077942.308919:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077942.308920:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077942.308924:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4631 00000100:00080000:0.0:1769077942.308927:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077942.308929:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077942.308932:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077942.308939:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923925632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077942.308954:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1769077942.308961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077942.308964:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077942.308968:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077942.308973:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4631 00000100:00080000:0.0:1769077942.308979:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077942.308984:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077942.308997:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077942.309003:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077942/real 1769077942] req@ffff969e06a60000 x1855007923925248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077942.309020:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923925760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077942.309029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923925248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077942.309042:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077942.309048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077942.309053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923925248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077942.309070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077942.309096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077942.309127:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077942/real 1769077942] req@ffff969e06a60a80 x1855007923925376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077942.309142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923925376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077942.309151:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077942.309155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077942.309158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077942.309182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077942.309205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077942.309217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077942/real 1769077942] req@ffff969e06a63800 x1855007923925504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077942.309230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923925504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077942.309239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077942.309243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077942.309246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077942.309253:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077942.309273:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077942.309284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077942/real 1769077942] req@ffff969e06a60700 x1855007923925632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077942.309305:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923925632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077942.309314:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077942.309317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077942.309321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077942.309327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077942.309348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077942.309359:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077942/real 1769077942] req@ffff969e06a62d80 x1855007923925760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077942.309373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923925760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077942.309382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077942.309385:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077942.309389:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923925760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077942.628665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077943.332798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077943.652701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077944.356762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077944.676715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077945.380748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077945.700741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077946.404732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077946.724730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077947.428715:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077947.428729:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077947.428734:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1769077947.428739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077947.428746:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4636 00000100:00080000:0.0:1769077947.428760:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077947.428769:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077947.428807:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923925888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077947.428839:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077947.428844:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077947.428846:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077947.428851:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4636 00000100:00080000:0.0:1769077947.428856:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077947.428860:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077947.428871:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077947.428885:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923926016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077947.428893:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923925888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077947.428902:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077947.428908:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077947.428911:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077947.428917:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4636 00000100:00080000:0.0:1769077947.428937:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077947.428943:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077947.428949:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077947.428957:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077947.428966:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923926144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077947.428979:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077947.428990:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077947/real 1769077947] req@ffff969e06a61500 x1855007923925888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077947.428995:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077947.429000:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077947.429007:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4636 00000100:00100000:3.0:1769077947.429012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923925888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077947.429013:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077947.429022:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077947.429028:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:3.0:1769077947.429040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077947.429045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077947.429049:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923926272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077947.429051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923925888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077947.429066:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077947.429070:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077947.429073:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077947.429076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926016:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769077947.429082:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4636 00000100:00080000:0.0:1769077947.429088:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077947.429093:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077947.429099:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:3.0:1769077947.429108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077947.429124:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923926400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077947.429126:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077947/real 1769077947] req@ffff969e06a60380 x1855007923926016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077947.429155:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923926016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077947.429173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077947.429179:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077947.429184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077947.429190:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077947.429204:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077947.429211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077947/real 1769077947] req@ffff969e06a62680 x1855007923926144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077947.429219:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923926144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077947.429224:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077947.429225:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077947.429227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077947.429236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077947.429250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077947.429257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077947/real 1769077947] req@ffff969e06a61f80 x1855007923926272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077947.429264:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923926272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077947.429270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077947.429271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077947.429273:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077947.429284:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077947.429294:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077947.429300:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077947/real 1769077947] req@ffff969e06a61180 x1855007923926400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077947.429307:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923926400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077947.429312:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077947.429314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077947.429316:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077947.748767:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077948.452759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077948.772748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077949.476782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077949.796681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077950.500710:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077950.820699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077951.524763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077951.844669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769077952.548701:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077952.548711:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077952.548716:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769077952.548737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077952.548742:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4641 00000100:00080000:1.0:1769077952.548753:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077952.548760:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769077952.548796:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923926528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077952.548824:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077952.548828:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077952.548830:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077952.548836:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4641 00000100:00080000:1.0:1769077952.548839:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077952.548843:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077952.548852:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077952.548866:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923926656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077952.548868:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923926528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769077952.548881:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077952.548884:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077952.548886:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077952.548889:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4641 00000100:00080000:1.0:1769077952.548892:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077952.548894:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077952.548897:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077952.548914:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923926784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077952.548931:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077952.548934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1769077952.548938:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769077952.548939:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077952.548947:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4641 00000100:00080000:1.0:1769077952.548951:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077952.548955:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077952.548960:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769077952.548970:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923926912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077952.548975:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077952/real 1769077952] req@ffff969e10b60380 x1855007923926528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769077952.548979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077952.548981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769077952.548983:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077952.548987:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4641 00000100:00080000:1.0:1769077952.548993:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077952.548999:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769077952.549007:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077952.549014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923926528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769077952.549021:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923927040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077952.549025:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077952.549033:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077952.549041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923926528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077952.549062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077952.549079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077952.549088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077952/real 1769077952] req@ffff969e10b61180 x1855007923926656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077952.549096:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923926656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077952.549102:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077952.549104:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077952.549107:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077952.549113:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077952.549133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077952.549144:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077952/real 1769077952] req@ffff969e10b60000 x1855007923926784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077952.549158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923926784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077952.549178:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077952.549181:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077952.549183:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077952.549188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077952.549205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077952.549212:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077952/real 1769077952] req@ffff969e10b62d80 x1855007923926912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077952.549220:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923926912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077952.549226:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077952.549237:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077952.549239:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923926912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077952.549243:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077952.549255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077952.549261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077952/real 1769077952] req@ffff969e10b60700 x1855007923927040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077952.549269:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923927040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077952.549275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077952.549276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077952.549283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077952.868731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077953.572845:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077953.892657:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077954.596735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077954.916745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077955.620800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077955.940746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077956.644729:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077956.964729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077957.668725:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077957.668735:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077957.668740:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077957.668751:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4646 00000100:00080000:2.0:1769077957.668762:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077957.668769:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077957.668794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077957.668814:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923927168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077957.668839:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077957.668841:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077957.668843:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077957.668846:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4646 00000100:00080000:2.0:1769077957.668866:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077957.668871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923927168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077957.668873:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077957.668886:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077957.668897:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923927296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077957.668906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077957.668908:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077957.668910:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077957.668912:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4646 00000100:00080000:2.0:1769077957.668922:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077957.668925:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077957.668929:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077957.668938:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923927424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077957.668945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077957.668948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077957.668949:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077957.668951:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4646 00000100:00080000:2.0:1769077957.668954:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077957.668959:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:3.0:1769077957.668960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769077957.668967:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077957.668976:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923927552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077957.668993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077957.668996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1769077957.668999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077957/real 1769077957] req@ffff969e0a46d500 x1855007923927168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077957.669000:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077957.669009:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4646 00000100:00080000:2.0:1769077957.669017:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077957.669022:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077957.669026:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769077957.669028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923927168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077957.669040:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769077957.669045:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923927680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077957.669047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077957.669053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923927168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769077957.669083:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077957.669099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077957.669108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077957/real 1769077957] req@ffff969e0a46ea00 x1855007923927296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077957.669116:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923927296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077957.669122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077957.669124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077957.669126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077957.669129:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:3.0:1769077957.669152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077957.669158:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077957/real 1769077957] req@ffff969e0a46f480 x1855007923927424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077957.669166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923927424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077957.669171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077957.669173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077957.669175:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077957.669178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077957.669189:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077957.669194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077957/real 1769077957] req@ffff969e0a46f800 x1855007923927552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077957.669201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923927552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077957.669207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077957.669208:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077957.669210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077957.669213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:3.0:1769077957.669222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769077957.669227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077957/real 1769077957] req@ffff969e0a46ce00 x1855007923927680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077957.669244:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923927680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077957.669250:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077957.669252:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769077957.669254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077957.988774:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077958.692816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077959.012730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077959.716849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077960.036742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077960.740790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077961.060731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077961.764786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077962.084729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077962.788710:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077962.788720:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077962.788724:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077962.788735:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4651 00000100:00080000:2.0:1769077962.788745:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077962.788752:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077962.788789:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923927808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769077962.788793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077962.788822:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077962.788825:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077962.788826:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077962.788829:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4651 00000100:00080000:2.0:1769077962.788832:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077962.788834:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077962.788859:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077962.788870:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923927808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077962.788873:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923927936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077962.788884:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077962.788887:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077962.788888:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077962.788892:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4651 00000100:00080000:2.0:1769077962.788895:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077962.788898:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077962.788902:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077962.788912:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923928064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077962.788918:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077962.788920:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077962.788921:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077962.788923:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4651 00000100:00080000:2.0:1769077962.788934:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077962.788937:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077962.788940:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077962.788948:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923928192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077962.788955:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077962.788957:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077962.788963:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077962.788967:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4651 00000100:00080000:2.0:1769077962.788970:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077962.788975:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769077962.788979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927936:192.168.204.143@tcp:38:kworker.0 00000008:00100000:2.0:1769077962.788981:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077962.789001:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923928320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077962.789026:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769077962.789045:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077962.789063:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077963.108707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077963.812813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077964.132774:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077964.836793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077965.156715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077965.860873:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1769077965.860982:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:3.0:1769077965.861028:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769077965.861047:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077965.861092:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077965.861110:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769077965.861118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077962/real 1769077965] req@ffff969e0a46c380 x1855007923927936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1769077965.861125:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769077965.861142:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:2.0:1769077965.861156:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923927936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077965.861172:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077965.861179:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077965.861184:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923927936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077965.861192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077962/real 1769077965] req@ffff969e0a46ce00 x1855007923928064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077965.861205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923928064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077965.861215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077965.861219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077965.861222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077965.861229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077962/real 1769077965] req@ffff969e0a46f800 x1855007923928192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077965.861267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923928192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077965.861276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077965.861281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077965.861284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077965.861293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077962/real 1769077965] req@ffff969e0a46d880 x1855007923927808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077965.861338:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923927808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077965.861347:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077965.861351:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077965.861355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923927808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077965.861390:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077962/real 1769077965] req@ffff969e0a46f480 x1855007923928320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077965.861403:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923928320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077965.861411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077965.861415:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077965.861418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077966.180718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077966.884908:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077967.204720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077967.908753:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077967.908781:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077967.908785:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077967.908797:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4657 00000100:00080000:2.0:1769077967.908810:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077967.908817:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077967.908865:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923928448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769077967.908911:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077967.908921:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077967.908927:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077967.908930:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077967.908936:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4657 00000100:00080000:2.0:1769077967.908941:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077967.908946:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077967.908963:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077967.908977:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923928576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077967.908988:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077967.908992:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077967.908994:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077967.908998:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4657 00000100:00080000:2.0:1769077967.909002:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077967.909006:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077967.909011:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077967.909022:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923928704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077967.909040:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077967.909044:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077967.909046:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077967.909050:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4657 00000100:00080000:2.0:1769077967.909055:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077967.909059:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077967.909064:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077967.909076:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923928832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077967.909085:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077967.909089:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077967.909091:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077967.909094:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4657 00000100:00080000:2.0:1769077967.909099:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077967.909103:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077967.909108:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077967.909119:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923928960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077967.909229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923928448:192.168.204.143@tcp:250:kworker.0 00000800:00000100:0.0:1769077967.909344:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077967.909380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077967/real 1769077967] req@ffff969e0a46f480 x1855007923928448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077967.909400:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923928448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077967.909411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077967.909416:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077967.909420:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923928448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077967.909428:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077967.909453:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077967.909466:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077967/real 1769077967] req@ffff969e0a46d880 x1855007923928576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077967.909479:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923928576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077967.909502:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077967.909505:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077967.909509:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077967.909516:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077967.909540:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077967.909552:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077967/real 1769077967] req@ffff969e0a46f800 x1855007923928704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077967.909563:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923928704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077967.909570:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077967.909574:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077967.909577:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077967.909583:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077967.909631:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077967.909643:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077967/real 1769077967] req@ffff969e0a46ce00 x1855007923928832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077967.909657:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923928832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077967.909666:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077967.909669:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077967.909680:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077967.909687:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077967.909712:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077967.909725:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077967/real 1769077967] req@ffff969e0a46c380 x1855007923928960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077967.909738:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923928960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077967.909747:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077967.909751:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077967.909756:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923928960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077968.228715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077968.932821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077969.252706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077969.956785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077970.276708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077970.980795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077971.300689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077972.004744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077972.324700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077973.028705:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077973.028718:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077973.028724:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077973.028738:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4662 00000100:00080000:0.0:1769077973.028751:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077973.028760:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077973.028783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077973.028812:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923929088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077973.028848:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077973.028852:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077973.028855:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077973.028862:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4662 00000100:00080000:0.0:1769077973.028875:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769077973.028887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923929088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077973.028906:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077973.028921:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077973.028939:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923929216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077973.028954:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769077973.028961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077973.028967:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077973.028971:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077973.028981:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4662 00000100:00080000:0.0:1769077973.028989:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077973.028994:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077973.029001:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077973.029001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077973/real 1769077973] req@ffff969e06a63100 x1855007923929088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077973.029030:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923929344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077973.029054:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077973.029060:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077973.029062:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077973.029069:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923929088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077973.029070:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4662 00000100:00080000:0.0:1769077973.029083:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077973.029088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077973.029095:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:2.0:1769077973.029099:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077973.029107:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923929088:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769077973.029108:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077973.029132:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923929472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077973.029145:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929216:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769077973.029148:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077973.029161:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077973.029164:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077973.029171:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4662 00000100:00080000:0.0:1769077973.029196:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077973.029204:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077973.029210:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769077973.029215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077973.029234:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923929600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769077973.029245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077973/real 1769077973] req@ffff969e06a61880 x1855007923929216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077973.029267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923929216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077973.029276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077973.029281:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077973.029285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077973.029293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077973.029317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077973.029329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077973/real 1769077973] req@ffff969e06a61180 x1855007923929344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077973.029342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923929344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077973.029365:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077973.029369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077973.029373:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077973.029389:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077973.029410:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077973.029420:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077973/real 1769077973] req@ffff969e06a61f80 x1855007923929472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077973.029433:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923929472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077973.029442:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077973.029445:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077973.029448:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077973.029455:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077973.029469:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077973.029476:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077973/real 1769077973] req@ffff969e06a62680 x1855007923929600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077973.029483:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923929600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077973.029489:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077973.029491:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077973.029499:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077973.348730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077974.052716:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077974.372712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077975.076790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077975.396708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077976.100725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077976.420738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077977.124789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077977.444707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077978.148727:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077978.148743:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077978.148749:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077978.148762:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4667 00000100:00080000:2.0:1769077978.148777:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077978.148786:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077978.148802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077978.148840:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923929728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077978.148879:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077978.148883:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077978.148885:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077978.148890:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4667 00000100:00080000:2.0:1769077978.148894:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077978.148900:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077978.148912:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077978.148926:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923929856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077978.148931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923929728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077978.148946:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077978.148950:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077978.148952:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077978.148958:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4667 00000100:00080000:2.0:1769077978.148962:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077978.148996:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077978.149003:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077978.149007:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077978.149022:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923929984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077978.149035:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077978.149040:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077978.149042:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077978.149044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077978/real 1769077978] req@ffff969e0a46ea00 x1855007923929728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077978.149052:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4667 00000100:00080000:2.0:1769077978.149062:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077978.149066:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923929728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077978.149069:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077978.149081:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769077978.149084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077978.149090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077978.149106:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923929728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769077978.149116:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923930112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077978.149129:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077978.149133:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1769077978.149134:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929856:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769077978.149136:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077978.149144:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4667 00000100:00080000:2.0:1769077978.149149:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077978.149154:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:0.0:1769077978.149159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769077978.149164:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077978.149171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077978/real 1769077978] req@ffff969e0a46d500 x1855007923929856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077978.149182:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923930240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077978.149185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923929856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077978.149205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077978.149210:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077978.149215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077978.149223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077978.149247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077978.149259:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077978/real 1769077978] req@ffff969e0a46ca80 x1855007923929984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077978.149270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923929984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077978.149276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077978.149279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077978.149281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923929984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077978.149286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077978.149300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077978.149307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077978/real 1769077978] req@ffff969e0a46df80 x1855007923930112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077978.149315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923930112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077978.149320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077978.149322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077978.149330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077978.149343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077978.149357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077978.149363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077978/real 1769077978] req@ffff969e0a46f100 x1855007923930240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077978.149371:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923930240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077978.149377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077978.149380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077978.149382:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077978.468717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077979.172835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077979.492727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077980.196774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077980.516741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077981.220779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077981.540724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077982.244787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077982.564725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077983.268713:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077983.268744:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077983.268749:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077983.268759:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4672 00000100:00080000:2.0:1769077983.268772:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077983.268779:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077983.268801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077983.268816:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923930368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077983.268839:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077983.268841:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077983.268846:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077983.268849:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4672 00000100:00080000:2.0:1769077983.268851:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077983.268854:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077983.268863:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077983.268880:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923930496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077983.268885:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923930368:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077983.268897:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077983.268901:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077983.268903:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077983.268908:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4672 00000100:00080000:2.0:1769077983.268912:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077983.268916:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077983.268921:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077983.268934:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923930624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077983.268945:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077983.268947:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077983.268948:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077983.268952:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4672 00000800:00000100:0.0:1769077983.268955:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769077983.268957:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077983.268962:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077983.268966:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077983.268974:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923930752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077983.268979:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077983.268982:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077983.268983:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077983.268986:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077983/real 1769077983] req@ffff969e0a46e300 x1855007923930368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077983.268988:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4672 00000100:00080000:2.0:1769077983.268993:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077983.268998:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077983.269002:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077983.269023:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923930880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077983.269029:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923930368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077983.269039:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077983.269043:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077983.269046:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923930368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077983.269066:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077983.269082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077983.269097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077983/real 1769077983] req@ffff969e0a46ed80 x1855007923930496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077983.269105:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923930496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077983.269110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077983.269112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077983.269114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077983.269118:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077983.269132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077983.269141:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077983/real 1769077983] req@ffff969e0a46f100 x1855007923930624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077983.269152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923930624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077983.269160:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077983.269163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077983.269166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077983.269171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077983.269185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077983.269191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077983/real 1769077983] req@ffff969e0a46df80 x1855007923930752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077983.269197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923930752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077983.269207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077983.269209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077983.269211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077983.269215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077983.269226:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077983.269232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077983/real 1769077983] req@ffff969e0a46ca80 x1855007923930880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077983.269238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923930880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077983.269243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077983.269244:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077983.269246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923930880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077983.588696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077984.292808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077984.612704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077985.316773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077985.636744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077986.340822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077986.660691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077987.365233:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077987.684682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077988.388735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077988.388750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077988.388756:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077988.388767:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4677 00000100:00080000:2.0:1769077988.388800:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077988.388807:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769077988.388841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077988.388848:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923931008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077988.388878:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077988.388881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077988.388883:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077988.388887:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4677 00000100:00080000:2.0:1769077988.388890:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077988.388894:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077988.388905:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077988.388916:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923931136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077988.388953:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923931008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769077988.388955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077988.388963:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077988.388965:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077988.388970:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4677 00000100:00080000:2.0:1769077988.388973:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077988.388977:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077988.388982:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077988.388993:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923931264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077988.389003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077988.389013:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077988.389015:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077988.389019:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4677 00000100:00080000:2.0:1769077988.389023:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077988.389028:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077988.389032:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769077988.389045:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077988.389045:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923931392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077988.389058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077988.389061:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077988.389063:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077988.389068:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4677 00000100:00080000:2.0:1769077988.389072:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077988.389076:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077988.389081:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769077988.389090:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077988/real 1769077988] req@ffff969e0a46d500 x1855007923931008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077988.389093:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923931520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077988.389131:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923931008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077988.389143:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077988.389148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077988.389152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923931008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077988.389185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077988.389222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077988.389233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077988/real 1769077988] req@ffff969e0a46ea00 x1855007923931136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077988.389245:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923931136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077988.389253:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077988.389256:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077988.389259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077988.389265:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077988.389282:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077988.389292:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077988/real 1769077988] req@ffff969e0a46c380 x1855007923931264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077988.389303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923931264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077988.389310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077988.389313:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077988.389317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077988.389322:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077988.389340:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077988.389349:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077988/real 1769077988] req@ffff969e0a46ce00 x1855007923931392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077988.389367:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923931392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077988.389374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077988.389377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077988.389381:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077988.389387:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077988.389405:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077988.389414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077988/real 1769077988] req@ffff969e0a46f800 x1855007923931520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077988.389425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923931520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077988.389433:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077988.389437:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077988.389440:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769077988.708699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077989.412793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077989.732675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077990.437286:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077990.756916:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077991.461268:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077991.780771:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077992.485011:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077992.804692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769077993.508798:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077993.508811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077993.508817:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077993.508828:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4682 00000100:00080000:2.0:1769077993.508905:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077993.508914:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769077993.508962:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923931648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077993.509020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077993.509032:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077993.509034:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077993.509039:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4682 00000100:00080000:2.0:1769077993.509043:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077993.509047:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077993.509058:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077993.509071:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923931776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077993.509080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077993.509083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077993.509084:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077993.509088:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4682 00000100:00080000:2.0:1769077993.509091:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077993.509095:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077993.509099:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077993.509111:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923931904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077993.509119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077993.509122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769077993.509123:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077993.509126:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4682 00000100:00080000:2.0:1769077993.509135:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077993.509139:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077993.509144:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077993.509154:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923932032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077993.509163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077993.509166:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769077993.509168:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077993.509170:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4682 00000100:00080000:2.0:1769077993.509173:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077993.509176:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769077993.509180:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077993.509187:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923932160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077993.509303:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923931648:192.168.204.143@tcp:250:kworker.0 00000800:00000100:0.0:1769077993.509379:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00010000:00010000:3.0:1769077993.509390:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077993.509418:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077993/real 1769077993] req@ffff969e0a46d880 x1855007923931648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077993.509436:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923931648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077993.509453:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077993.509457:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077993.509461:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923931648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077993.509474:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077993.509499:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077993.509510:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077993/real 1769077993] req@ffff969e0a46f480 x1855007923931776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077993.509521:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923931776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077993.509528:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077993.509531:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077993.509534:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077993.509539:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769077993.509558:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077993.509567:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077993/real 1769077993] req@ffff969e0a46f800 x1855007923931904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077993.509577:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923931904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077993.509584:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077993.509587:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077993.509616:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923931904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077993.509630:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077993.509654:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077993.509665:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077993/real 1769077993] req@ffff969e0a46ce00 x1855007923932032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077993.509677:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923932032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077993.509685:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077993.509689:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077993.509692:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769077993.509697:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769077993.509715:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769077993.509725:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077993/real 1769077993] req@ffff969e0a46c380 x1855007923932160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077993.509735:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923932160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077993.509743:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077993.509746:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769077993.509749:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077993.828710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077994.535959:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077994.852724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077995.557145:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077995.876722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077996.580832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077996.900741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077997.604835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077997.924689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769077998.628730:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077998.628741:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077998.628746:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077998.628797:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4687 00010000:00010000:3.0:1769077998.628805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077998.628809:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077998.628817:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769077998.628849:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923932288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077998.628877:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077998.628880:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077998.628882:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077998.628886:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4687 00000100:00080000:0.0:1769077998.628890:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077998.628894:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077998.628903:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077998.628906:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923932288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769077998.628916:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923932416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077998.628927:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077998.628931:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077998.628933:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077998.628936:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4687 00000100:00080000:0.0:1769077998.628941:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077998.628944:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077998.628948:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077998.628958:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923932544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077998.628960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769077998.628982:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077998.628985:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769077998.628987:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769077998.628991:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077998/real 1769077998] req@ffff969e06a60380 x1855007923932288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077998.628994:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4687 00000100:00080000:0.0:1769077998.629000:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077998.629006:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077998.629011:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077998.629011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923932288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077998.629023:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769077998.629026:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923932672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077998.629028:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077998.629035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923932288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769077998.629040:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077998.629047:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769077998.629050:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077998.629054:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4687 00000100:00080000:0.0:1769077998.629058:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077998.629061:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769077998.629066:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769077998.629068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769077998.629078:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923932800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769077998.629089:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077998.629100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077998/real 1769077998] req@ffff969e06a61500 x1855007923932416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077998.629108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923932416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077998.629113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077998.629116:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077998.629118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077998.629121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769077998.629131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077998.629136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077998/real 1769077998] req@ffff969e06a62d80 x1855007923932544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077998.629142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923932544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077998.629147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077998.629148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077998.629150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769077998.629153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077998.629162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077998.629167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077998/real 1769077998] req@ffff969e06a60700 x1855007923932672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077998.629180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923932672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077998.629186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077998.629187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077998.629189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769077998.629198:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769077998.629209:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769077998.629214:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077998/real 1769077998] req@ffff969e06a63800 x1855007923932800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769077998.629221:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923932800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769077998.629226:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769077998.629227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769077998.629228:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923932800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769077998.948710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077999.652779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077999.972675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078000.676698:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078000.996709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078001.700788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078002.020717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078002.724880:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078003.044735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078003.748752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078003.748763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078003.748767:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078003.748775:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4692 00000100:00080000:2.0:1769078003.748786:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078003.748793:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769078003.748794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078003.748827:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923932928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078003.748884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078003.748888:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078003.748891:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078003.748896:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4692 00000100:00080000:2.0:1769078003.748902:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078003.748907:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078003.748918:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078003.748935:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923932928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078003.748937:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923933056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078003.748963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078003.748968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078003.748971:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078003.748978:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4692 00000100:00080000:2.0:1769078003.748982:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078003.748986:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078003.748993:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078003.749008:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923933184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078003.749019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078003.749023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1769078003.749027:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078003.749028:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078003.749036:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4692 00000100:00080000:2.0:1769078003.749061:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078003.749064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078003/real 1769078003] req@ffff969e0a46ea00 x1855007923932928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078003.749075:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078003.749081:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078003.749095:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923933312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078003.749104:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923932928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078003.749107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078003.749119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078003.749120:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078003.749123:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078003.749127:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769078003.749139:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4692 00000100:00080000:2.0:1769078003.749145:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078003.749147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923932928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769078003.749153:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078003.749161:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078003.749173:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923933440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078003.749178:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078003.749220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078003.749234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078003/real 1769078003] req@ffff969e0a46d500 x1855007923933056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078003.749255:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923933056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078003.749261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078003.749263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078003.749265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078003.749269:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078003.749284:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078003.749291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078003/real 1769078003] req@ffff969e0a46ca80 x1855007923933184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078003.749298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923933184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078003.749313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078003.749315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078003.749317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078003.749322:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078003.749347:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078003.749357:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078003/real 1769078003] req@ffff969e0a46df80 x1855007923933312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078003.749369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923933312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078003.749377:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078003.749381:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078003.749384:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078003.749399:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078003.749412:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078003.749419:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078003/real 1769078003] req@ffff969e0a46f100 x1855007923933440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078003.749429:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923933440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078003.749436:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078003.749439:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078003.749449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078004.068737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078004.772840:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078005.092718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078005.796783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078006.116719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078006.820800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078007.140733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078007.844806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078008.164676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078008.868725:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078008.868736:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078008.868740:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078008.868750:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4698 00000100:00080000:2.0:1769078008.868760:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078008.868766:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078008.868803:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923933568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769078008.868803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078008.868836:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078008.868855:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078008.868856:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078008.868860:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4698 00000100:00080000:2.0:1769078008.868863:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078008.868868:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078008.868876:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078008.868888:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923933696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078008.868890:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923933568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769078008.868905:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078008.868908:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078008.868909:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078008.868912:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4698 00000100:00080000:2.0:1769078008.868914:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078008.868917:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078008.868920:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078008.868928:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923933824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078008.868935:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078008.868937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078008.868939:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078008.868942:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4698 00000100:00080000:2.0:1769078008.868944:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078008.868946:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078008.868954:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078008.868963:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923933952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078008.868971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078008.868972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078008.868974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078008.868978:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078008.868981:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4698 00000100:00080000:2.0:1769078008.868984:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078008.868987:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078008.868990:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078008.868997:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923934080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078008.869022:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078008/real 1769078008] req@ffff969e0a46ed80 x1855007923933568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078008.869045:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923933568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078008.869056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078008.869060:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078008.869065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923933568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078008.869092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078008.869119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078008.869133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078008/real 1769078008] req@ffff969e0a46e300 x1855007923933696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078008.869146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923933696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078008.869153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078008.869155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078008.869158:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078008.869172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078008.869188:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078008.869203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078008/real 1769078008] req@ffff969e0a46f100 x1855007923933824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078008.869212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923933824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078008.869218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078008.869220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078008.869223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078008.869226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078008.869239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078008.869245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078008/real 1769078008] req@ffff969e0a46df80 x1855007923933952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078008.869253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923933952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078008.869259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078008.869261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078008.869263:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923933952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078008.869268:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078008.869289:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078008.869300:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078008/real 1769078008] req@ffff969e0a46ca80 x1855007923934080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078008.869313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923934080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078008.869327:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078008.869332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078008.869336:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078009.188748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078009.892775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078010.212676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078010.916747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078011.236721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078011.940771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078012.260719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078012.964741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078013.284651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078013.988740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078013.988755:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078013.988762:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078013.988777:0:36:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4703 00000100:00080000:2.0:1769078013.988794:0:36:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078013.988832:0:36:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769078013.988866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078013.988891:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923934208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078013.988924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078013.988929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078013.988931:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078013.988936:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4703 00000100:00080000:2.0:1769078013.988978:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078013.988987:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078013.989002:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078013.989029:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923934336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078013.989052:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923934208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769078013.989054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078013.989068:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078013.989072:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078013.989082:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4703 00000100:00080000:2.0:1769078013.989091:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078013.989100:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078013.989111:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078013.989135:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923934464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078013.989148:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078013.989162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078013.989168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078013.989171:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078013.989178:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4703 00000100:00080000:2.0:1769078013.989187:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078013.989193:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078013.989201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078013/real 1769078013] req@ffff969e0a46d500 x1855007923934208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769078013.989205:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078013.989231:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923934592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078013.989237:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923934208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078013.989266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078013.989286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769078013.989308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078013.989316:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1769078013.989317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923934208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769078013.989323:0:36:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078013.989337:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4703 00000100:00080000:2.0:1769078013.989346:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078013.989357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934336:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769078013.989359:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078013.989376:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769078013.989397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078013.989405:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923934720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078013.989421:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078013/real 1769078013] req@ffff969e0a46ea00 x1855007923934336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078013.989439:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923934336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078013.989450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078013.989455:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078013.989459:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078013.989478:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078013.989565:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078013.989581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078013/real 1769078013] req@ffff969e0a46c380 x1855007923934464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078013.989597:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923934464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078013.989664:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078013.989669:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078013.989674:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078013.989682:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078013.989709:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078013.989723:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078013/real 1769078013] req@ffff969e0a46ce00 x1855007923934592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078013.989738:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923934592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078013.989749:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078013.989753:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078013.989757:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078013.989779:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078013.989798:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078013.989809:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078013/real 1769078013] req@ffff969e0a46f800 x1855007923934720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078013.989842:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923934720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078013.989853:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078013.989857:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078013.989861:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078014.308709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078015.012714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078015.332686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078016.036813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078016.356719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078017.060775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078017.380704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078018.084771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078018.404723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078019.108721:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078019.108733:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078019.108738:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078019.108750:0:32130:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4708 00000100:00080000:0.0:1769078019.108764:0:32130:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769078019.108771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078019.108772:0:32130:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078019.108814:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923934848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078019.108844:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078019.108848:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078019.108850:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078019.108855:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4708 00000100:00080000:0.0:1769078019.108859:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078019.108873:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078019.108879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923934848:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078019.108889:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078019.108907:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923934976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078019.108941:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078019.108945:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078019.108948:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078019.108953:0:32130:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4708 00000100:00080000:0.0:1769078019.108958:0:32130:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769078019.108962:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078019.108966:0:32130:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078019.108974:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078019.108990:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923935104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078019.108999:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078019/real 1769078019] req@ffff969e06a60a80 x1855007923934848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078019.109004:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078019.109012:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078019.109015:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078019.109021:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4708 00000100:00100000:2.0:1769078019.109024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923934848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078019.109027:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078019.109036:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078019.109046:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:2.0:1769078019.109046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078019.109056:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078019.109063:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923935232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078019.109065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923934848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078019.109083:0:32130:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078019.109087:0:32130:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078019.109089:0:32130:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078019.109095:0:32130:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4708 00000100:00100000:2.0:1769078019.109098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934976:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078019.109107:0:32130:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078019.109114:0:32130:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078019.109121:0:32130:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078019.109139:0:32130:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923935360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078019.109149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078019.109171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078019/real 1769078019] req@ffff969e06a60000 x1855007923934976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078019.109195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923934976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078019.109205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078019.109209:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078019.109213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923934976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078019.109220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078019.109243:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078019.109255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078019/real 1769078019] req@ffff969e06a63800 x1855007923935104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078019.109268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923935104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078019.109277:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078019.109280:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078019.109284:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078019.109290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078019.109309:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078019.109319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078019/real 1769078019] req@ffff969e06a60700 x1855007923935232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078019.109332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923935232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078019.109358:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078019.109362:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078019.109366:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078019.109383:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078019.109431:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078019.109442:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078019/real 1769078019] req@ffff969e06a62d80 x1855007923935360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078019.109456:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923935360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078019.109465:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078019.109469:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078019.109473:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078019.428692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078020.132783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078020.452735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078021.156788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078021.476702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078022.180771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078022.500701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078023.204811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078023.524720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078024.228717:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078024.228730:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078024.228734:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078024.228746:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4713 00000100:00080000:0.0:1769078024.228758:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078024.228767:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769078024.228790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078024.228802:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923935488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078024.228828:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078024.228832:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078024.228834:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078024.228839:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4713 00000100:00080000:0.0:1769078024.228844:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078024.228849:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078024.228859:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078024.228873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923935488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078024.228877:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923935616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078024.228888:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078024.228905:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078024.228907:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078024.228911:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4713 00000100:00080000:0.0:1769078024.228914:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078024.228918:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078024.228922:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078024.228933:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923935744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078024.228937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078024.228946:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078024.228949:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078024.228951:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078024.228955:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4713 00000100:00080000:0.0:1769078024.228958:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078024.228961:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078024.228965:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078024.228973:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078024/real 1769078024] req@ffff969e06a61500 x1855007923935488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078024.228975:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923935872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078024.229012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923935488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078024.229015:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078024.229024:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078024.229028:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078024.229031:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078024.229033:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4713 00000100:00080000:0.0:1769078024.229040:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078024.229042:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769078024.229049:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078024.229053:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923935488:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078024.229063:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078024.229082:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923936000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078024.229095:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078024.229119:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078024.229129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078024/real 1769078024] req@ffff969e06a60380 x1855007923935616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078024.229138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923935616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078024.229145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078024.229148:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078024.229150:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078024.229155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078024.229166:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078024.229172:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078024/real 1769078024] req@ffff969e06a62680 x1855007923935744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078024.229180:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923935744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078024.229185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078024.229187:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078024.229189:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078024.229192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078024.229203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078024.229208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078024/real 1769078024] req@ffff969e06a61f80 x1855007923935872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078024.229222:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923935872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078024.229230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078024.229233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078024.229237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923935872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078024.229255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078024.229278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078024.229290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078024/real 1769078024] req@ffff969e06a61180 x1855007923936000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078024.229303:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923936000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078024.229313:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078024.229317:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078024.229349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078024.548736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078025.252784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078025.572685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078026.276754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078026.596726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078027.300823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078027.620651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078028.324767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078028.644709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078029.348726:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078029.348738:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078029.348745:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078029.348758:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4718 00010000:00010000:0.0:1769078029.348769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078029.348771:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078029.348799:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078029.348835:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923936128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078029.348865:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078029.348872:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078029.348874:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078029.348881:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4718 00000100:00080000:3.0:1769078029.348886:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078029.348890:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078029.348900:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078029.348916:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923936128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078029.348918:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923936256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078029.348936:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078029.348940:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078029.348943:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078029.348947:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4718 00000100:00080000:3.0:1769078029.348953:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078029.348957:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078029.348961:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078029.348972:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923936384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078029.348979:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078029.348982:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078029.348988:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078029.348992:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4718 00000100:00080000:3.0:1769078029.348995:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078029.348999:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078029.349005:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078029.349018:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923936512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078029.349030:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078029.349033:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936256:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769078029.349046:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078029.349050:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078029.349055:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4718 00000100:00080000:3.0:1769078029.349066:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769078029.349075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936384:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769078029.349075:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078029.349083:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078029.349107:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923936640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078029.349125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078029.349161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078029.668672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078030.372742:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078030.692703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078031.396775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078031.716681:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078032.420798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078032.420912:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769078032.420954:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078032.420973:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078032.421009:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078032.421017:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078032.421028:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078032.421042:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:0.0:1769078032.421045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078029/real 1769078032] req@ffff969e0a7fc380 x1855007923936256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.421073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923936256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.421086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.421092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078032.421096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078032.421103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078029/real 1769078032] req@ffff969e0a7fc000 x1855007923936384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.421117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923936384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.421126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.421129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078032.421132:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078032.421138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078029/real 1769078032] req@ffff969e0a7ffb80 x1855007923936128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.421173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923936128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.421180:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.421216:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078032.421221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923936128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078032.421257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078029/real 1769078032] req@ffff969e0a7fdc00 x1855007923936640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.421268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923936640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.421276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.421280:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078032.421283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078032.421289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078029/real 1769078032] req@ffff969e0a7ff800 x1855007923936512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078032.421298:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923936512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078032.421305:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078032.421308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078032.421310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078032.740694:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078033.444820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078033.764742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078034.468708:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078034.468721:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078034.468726:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078034.468752:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4723 00010000:00010000:1.0:1769078034.468754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078034.468763:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078034.468771:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078034.468812:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923936768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078034.468846:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078034.468850:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078034.468851:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078034.468856:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4723 00000100:00080000:3.0:1769078034.468860:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078034.468865:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078034.468876:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078034.468886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923936768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078034.468889:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923936896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078034.468906:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078034.468909:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078034.468911:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078034.468915:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4723 00000100:00080000:3.0:1769078034.468919:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078034.468923:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078034.468941:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078034.468953:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923937024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078034.468979:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078034.468979:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078034.468985:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078034.468988:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078034.468993:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4723 00000100:00080000:3.0:1769078034.468998:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078034.469003:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078034.469009:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078034.469026:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078034/real 1769078034] req@ffff969e0a7fd500 x1855007923936768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078034.469028:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923937152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078034.469049:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078034.469053:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078034.469055:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078034.469058:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923936768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078034.469062:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4723 00000100:00080000:3.0:1769078034.469068:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078034.469079:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769078034.469084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078034.469086:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769078034.469093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078034.469099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923936768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078034.469102:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923937280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078034.469132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078034.469170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078034.469183:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078034/real 1769078034] req@ffff969e0a7fd880 x1855007923936896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078034.469194:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923936896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078034.469202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078034.469205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078034.469208:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923936896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078034.469214:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078034.469229:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078034.469237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078034/real 1769078034] req@ffff969e0a7fd180 x1855007923937024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078034.469258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923937024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078034.469266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078034.469269:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078034.469272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078034.469277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078034.469297:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078034.469306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078034/real 1769078034] req@ffff969e0a7fe300 x1855007923937152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078034.469317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923937152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078034.469324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078034.469327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078034.469330:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078034.469335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078034.469351:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078034.469359:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078034/real 1769078034] req@ffff969e0a7fca80 x1855007923937280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078034.469369:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923937280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078034.469387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078034.469392:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078034.469395:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078034.788678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078035.492805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078035.812688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078036.516749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078036.836695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078037.540723:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078037.860724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078038.564712:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078038.884703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078039.588686:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078039.588694:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078039.588698:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078039.588706:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4728 00000100:00080000:3.0:1769078039.588716:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1769078039.588719:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078039.588722:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078039.588759:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923937408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078039.588781:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078039.588783:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078039.588785:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078039.588787:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4728 00000100:00080000:3.0:1769078039.588790:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078039.588792:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078039.588799:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078039.588806:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923937536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078039.588813:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078039.588816:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078039.588817:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078039.588819:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4728 00000100:00080000:3.0:1769078039.588821:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078039.588823:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078039.588825:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078039.588841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923937408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078039.588852:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923937664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078039.588860:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078039.588862:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078039.588863:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078039.588866:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4728 00000100:00080000:3.0:1769078039.588869:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078039.588871:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078039.588874:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078039.588883:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923937792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078039.588888:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078039.588891:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078039.588892:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078039.588893:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4728 00000100:00080000:3.0:1769078039.588895:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078039.588897:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078039.588899:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078039.588905:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923937920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078039.588916:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078039.588949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078039/real 1769078039] req@ffff969e0a7fdf80 x1855007923937408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078039.588962:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923937408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078039.588969:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078039.588972:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078039.588974:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923937408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078039.588979:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078039.588994:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078039.589002:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078039/real 1769078039] req@ffff969e0a7fea00 x1855007923937536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078039.589009:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923937536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078039.589014:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078039.589016:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078039.589018:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078039.589042:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078039.589055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078039.589078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078039/real 1769078039] req@ffff969e0a7fc700 x1855007923937664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078039.589087:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923937664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078039.589092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078039.589098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078039.589100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078039.589104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078039.589118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078039.589124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078039/real 1769078039] req@ffff969e0a7fce00 x1855007923937792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078039.589130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923937792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078039.589136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078039.589138:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078039.589140:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078039.589143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078039.589151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078039.589155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078039/real 1769078039] req@ffff969e0a7fe680 x1855007923937920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078039.589168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923937920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078039.589173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078039.589175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078039.589177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923937920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078039.908720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078040.612765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078040.932686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078041.636718:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078041.956707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078042.660668:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078042.980693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078043.684736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078044.004696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078044.708719:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1769078044.708725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078044.708730:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078044.708735:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078044.708744:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4733 00000100:00080000:0.0:1769078044.708754:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078044.708759:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078044.708789:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923938048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078044.708814:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078044.708816:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078044.708818:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078044.708820:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4733 00000100:00080000:0.0:1769078044.708823:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078044.708825:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078044.708831:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078044.708840:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923938176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078044.708859:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923938048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078044.708872:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078044.708874:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078044.708875:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078044.708879:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4733 00000100:00080000:0.0:1769078044.708882:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078044.708884:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078044.708887:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078044.708898:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923938304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078044.708903:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078044.708906:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078044.708907:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078044.708909:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4733 00000100:00080000:0.0:1769078044.708911:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078044.708913:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078044.708916:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:2.0:1769078044.708921:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078044.708925:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923938432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078044.708931:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078044.708933:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078044.708935:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078044.708937:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4733 00000100:00080000:0.0:1769078044.708943:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078044.708947:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078044.708951:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078044.708953:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078044/real 1769078044] req@ffff969e06a61880 x1855007923938048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078044.708960:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923938560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078044.708990:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923938048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078044.709001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078044.709006:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078044.709010:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923938048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078044.709032:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078044.709048:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078044.709056:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078044/real 1769078044] req@ffff969e06a63100 x1855007923938176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078044.709064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923938176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078044.709079:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078044.709082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078044.709084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078044.709088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078044.709100:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078044.709106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078044/real 1769078044] req@ffff969e06a61180 x1855007923938304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078044.709112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923938304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078044.709117:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078044.709119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078044.709120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078044.709123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078044.709133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078044.709138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078044/real 1769078044] req@ffff969e06a61f80 x1855007923938432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078044.709144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923938432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078044.709148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078044.709150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078044.709156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078044.709160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078044.709171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078044.709176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078044/real 1769078044] req@ffff969e06a62680 x1855007923938560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078044.709183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923938560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078044.709187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078044.709189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078044.709190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078045.028678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078045.732696:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078046.052711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078046.756746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078047.076667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078047.780722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078048.100741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078048.804724:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078049.124654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078049.828731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078049.828732:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078049.828740:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078049.828745:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078049.828755:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4739 00000100:00080000:3.0:1769078049.828766:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078049.828773:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078049.828805:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923938688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078049.828830:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078049.828834:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078049.828835:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078049.828840:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4739 00000100:00080000:3.0:1769078049.828843:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078049.828847:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078049.828856:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078049.828871:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923938816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078049.828877:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923938688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078049.828886:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078049.828905:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078049.828920:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078049.828925:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4739 00000100:00080000:3.0:1769078049.828930:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078049.828934:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078049.828940:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078049.828953:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923938944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078049.828963:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078049.828967:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078049.828969:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769078049.828974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078049.828978:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4739 00000100:00080000:3.0:1769078049.828982:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078049.828987:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078049.828992:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078049.829004:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923939072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078049.829007:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078049/real 1769078049] req@ffff969e0a7fed80 x1855007923938688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078049.829035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923938688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078049.829037:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078049.829045:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078049.829048:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078049.829050:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078049.829054:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4739 00000100:00080000:2.0:1769078049.829059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769078049.829060:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769078049.829065:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923938688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078049.829068:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078049.829076:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078049.829096:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:3.0:1769078049.829096:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923939200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078049.829122:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078049.829138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078049/real 1769078049] req@ffff969e0a7fe680 x1855007923938816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078049.829146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923938816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078049.829152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078049.829154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078049.829156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078049.829160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078049.829172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078049.829177:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078049/real 1769078049] req@ffff969e0a7fce00 x1855007923938944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078049.829189:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923938944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078049.829196:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078049.829200:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078049.829203:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923938944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078049.829209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078049.829228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078049.829238:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078049/real 1769078049] req@ffff969e0a7fc700 x1855007923939072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078049.829250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923939072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078049.829267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078049.829271:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078049.829275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078049.829290:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078049.829310:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078049.829320:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078049/real 1769078049] req@ffff969e0a7fea00 x1855007923939200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078049.829331:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923939200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078049.829340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078049.829343:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078049.829346:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078050.148699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078050.852752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078051.172709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078051.876707:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078052.196708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078052.900823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078053.220738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078053.924759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078054.244709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078054.948732:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078054.948743:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078054.948748:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078054.948760:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4744 00010000:00010000:0.0:1769078054.948772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078054.948772:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078054.948782:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078054.948842:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923939328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078054.948873:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078054.948896:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078054.948898:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078054.948905:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4744 00000100:00080000:2.0:1769078054.948910:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078054.948920:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078054.948922:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923939328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769078054.948933:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078054.948952:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923939456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078054.948962:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078054.948964:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078054.948967:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078054.948970:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4744 00000100:00080000:2.0:1769078054.948973:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078054.948976:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078054.948980:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078054.948994:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923939584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078054.949001:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078054.949010:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078054.949014:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078054.949016:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078054.949022:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4744 00000100:00080000:2.0:1769078054.949029:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078054.949036:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078054.949045:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078054.949051:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078054/real 1769078054] req@ffff969e0a46f480 x1855007923939328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078054.949058:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923939712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078054.949091:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078054.949095:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078054.949096:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078054.949100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923939328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078054.949106:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4744 00000100:00080000:2.0:1769078054.949116:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078054.949127:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769078054.949128:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078054.949139:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769078054.949143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078054.949152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923939328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078054.949157:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923939840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078054.949181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078054.949202:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078054.949210:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078054/real 1769078054] req@ffff969e0a46d880 x1855007923939456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078054.949238:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923939456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078054.949244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078054.949247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078054.949249:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078054.949254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078054.949267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078054.949274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078054/real 1769078054] req@ffff969e0a46f800 x1855007923939584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078054.949282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923939584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078054.949287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078054.949289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078054.949290:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078054.949294:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078054.949305:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078054.949311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078054/real 1769078054] req@ffff969e0a46ce00 x1855007923939712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078054.949318:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923939712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078054.949334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078054.949337:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078054.949339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078054.949343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078054.949357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078054.949364:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078054/real 1769078054] req@ffff969e0a46c380 x1855007923939840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078054.949370:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923939840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078054.949376:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078054.949377:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078054.949379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923939840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078055.268707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078055.972836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078056.292702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078056.997652:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078057.316708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078058.020854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078058.340712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078059.045114:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078059.364695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078060.068734:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078060.068753:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078060.068759:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078060.068776:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4749 00000100:00080000:2.0:1769078060.068794:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078060.068807:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078060.068867:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923939968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769078060.068872:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078060.068918:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078060.068923:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078060.068925:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078060.068930:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4749 00000100:00080000:2.0:1769078060.068950:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078060.068955:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078060.068969:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078060.068987:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923940096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078060.068999:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078060.069003:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078060.069005:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078060.069010:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4749 00000100:00080000:2.0:1769078060.069014:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078060.069018:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078060.069024:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078060.069048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923939968:192.168.204.143@tcp:250:kworker.0 00000800:00000100:0.0:1769078060.069139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078060.069181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078060/real 1769078060] req@ffff969e0a46ea00 x1855007923939968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078060.069204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923939968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078060.069222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078060.069227:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078060.069232:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923939968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078060.069240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078060.069267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078060.069281:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078060/real 1769078060] req@ffff969e0a46d500 x1855007923940096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078060.069293:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923940096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078060.069302:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078060.069306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078060.069309:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078060.069360:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923940224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078060.069390:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078060.069394:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078060.069397:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078060.069401:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4749 00000100:00080000:2.0:1769078060.069406:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078060.069419:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078060.069423:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940224:192.168.204.143@tcp:38:kworker.0 00000008:00100000:2.0:1769078060.069428:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078060.069448:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923940352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078060.069456:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078060.069473:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078060/real 1769078060] req@ffff969e0a46ca80 x1855007923940224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078060.069479:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078060.069489:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078060.069493:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078060.069500:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4749 00000100:00100000:0.0:1769078060.069502:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923940224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078060.069505:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078060.069514:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769078060.069519:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078060.069522:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769078060.069528:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078060.069534:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078060.069541:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923940480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078060.069562:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078060.069588:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078060.069646:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078060/real 1769078060] req@ffff969e0a46df80 x1855007923940352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078060.069660:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923940352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078060.069669:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078060.069673:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078060.069676:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078060.069681:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078060.069697:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078060.069705:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078060/real 1769078060] req@ffff969e0a46f100 x1855007923940480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078060.069712:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923940480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078060.069736:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078060.069740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078060.069743:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078060.388690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078061.094097:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078061.412725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078062.117666:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078062.436705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078063.140832:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078063.460741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078064.164798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078064.484699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078065.188716:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078065.188728:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078065.188733:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078065.188746:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4754 00000100:00080000:2.0:1769078065.188759:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078065.188768:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769078065.188784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078065.188806:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923940608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078065.188828:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078065.188831:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078065.188832:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078065.188836:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4754 00000100:00080000:2.0:1769078065.188838:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078065.188843:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078065.188865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923940608:192.168.204.143@tcp:250:kworker.0 00000008:00100000:2.0:1769078065.188866:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078065.188880:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923940736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078065.188890:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078065.188892:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078065.188894:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078065.188897:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4754 00000100:00080000:2.0:1769078065.188901:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078065.188904:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078065.188908:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078065.188918:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923940864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078065.188925:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078065.188928:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078065.188929:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078065.188932:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4754 00000100:00080000:2.0:1769078065.188934:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078065.188939:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:0.0:1769078065.188940:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:2.0:1769078065.188945:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078065.188957:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923940992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078065.188968:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078065.188972:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078065.188981:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078065.188987:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078065/real 1769078065] req@ffff969e0a46e300 x1855007923940608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078065.188989:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4754 00000100:00080000:2.0:1769078065.188997:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078065.189005:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078065.189011:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078065.189018:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923940608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078065.189032:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923941120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078065.189034:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078065.189045:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078065.189051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923940608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078065.189072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078065.189092:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078065.189101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078065/real 1769078065] req@ffff969e0a46ed80 x1855007923940736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078065.189110:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923940736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078065.189116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078065.189119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078065.189123:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078065.189129:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078065.189149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078065.189160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078065/real 1769078065] req@ffff969e0a46f100 x1855007923940864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078065.189173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923940864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078065.189194:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078065.189198:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078065.189202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078065.189209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078065.189232:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078065.189243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078065/real 1769078065] req@ffff969e0a46df80 x1855007923940992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078065.189254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923940992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078065.189262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078065.189266:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078065.189269:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923940992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078065.189275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078065.189295:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078065.189307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078065/real 1769078065] req@ffff969e0a46ca80 x1855007923941120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078065.189319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923941120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078065.189329:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078065.189332:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078065.189352:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078065.508739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078066.212773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078066.532692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078067.236779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078067.556714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078068.260766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078068.580719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078069.284759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078069.604695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078070.308708:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078070.308718:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078070.308722:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078070.308730:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4759 00000100:00080000:3.0:1769078070.308760:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078070.308766:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769078070.308773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078070.308796:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923941248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078070.308819:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078070.308822:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078070.308823:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078070.308826:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4759 00000100:00080000:3.0:1769078070.308829:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078070.308831:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078070.308839:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078070.308847:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923941376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078070.308855:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078070.308860:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078070.308862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923941248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078070.308863:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078070.308871:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4759 00000100:00080000:3.0:1769078070.308874:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078070.308877:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078070.308880:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078070.308888:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923941504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078070.308894:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078070.308902:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078070.308903:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078070.308906:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4759 00000100:00080000:3.0:1769078070.308910:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078070.308912:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078070.308916:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078070.308926:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923941632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078070.308932:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078070.308935:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078070.308936:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078070.308942:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4759 00000800:00000100:0.0:1769078070.308944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078070.308946:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078070.308951:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078070.308954:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078070.308961:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923941760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078070.308991:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078070/real 1769078070] req@ffff969e0a7fdf80 x1855007923941248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078070.309015:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923941248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078070.309026:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078070.309031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078070.309035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923941248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078070.309063:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078070.309087:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078070.309100:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078070/real 1769078070] req@ffff969e0a7fca80 x1855007923941376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078070.309115:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923941376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078070.309125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078070.309129:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078070.309142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078070.309150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078070.309172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078070.309184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078070/real 1769078070] req@ffff969e0a7fe300 x1855007923941504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078070.309198:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923941504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078070.309207:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078070.309211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078070.309226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078070.309233:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078070.309258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078070.309270:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078070/real 1769078070] req@ffff969e0a7fd180 x1855007923941632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078070.309284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923941632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078070.309294:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078070.309298:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078070.309301:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078070.309308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078070.309327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078070.309348:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078070/real 1769078070] req@ffff969e0a7fd880 x1855007923941760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078070.309362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923941760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078070.309372:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078070.309375:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078070.309379:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923941760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078070.628733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078071.332720:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078071.652703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078072.356793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078072.676668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078073.380819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078073.700692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078074.404814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078074.724720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078075.428719:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078075.428730:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078075.428758:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078075.428767:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4764 00000100:00080000:3.0:1769078075.428778:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078075.428786:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769078075.428796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078075.428827:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923941888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078075.428850:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078075.428852:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078075.428854:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078075.428857:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4764 00000100:00080000:3.0:1769078075.428859:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078075.428862:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078075.428873:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078075.428886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923941888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078075.428887:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923942016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078075.428904:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078075.428907:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078075.428909:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078075.428913:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4764 00000100:00080000:3.0:1769078075.428917:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078075.428922:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078075.428928:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078075.428943:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923942144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078075.428968:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078075.428973:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1769078075.428976:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078075.428977:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078075.428986:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4764 00000100:00080000:3.0:1769078075.428992:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078075.428998:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078075.429003:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078075.429021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078075/real 1769078075] req@ffff969e0a7fd500 x1855007923941888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078075.429021:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923942272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078075.429044:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078075.429050:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078075.429053:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078075.429058:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4764 00000100:00100000:0.0:1769078075.429062:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923941888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078075.429064:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078075.429074:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769078075.429082:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078075.429083:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769078075.429092:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078075.429097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923941888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078075.429103:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923942400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078075.429127:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078075.429149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078075.429168:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078075/real 1769078075] req@ffff969e0a7ff800 x1855007923942016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078075.429181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923942016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078075.429190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078075.429193:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078075.429197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078075.429203:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078075.429220:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078075.429229:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078075/real 1769078075] req@ffff969e0a7fdc00 x1855007923942144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078075.429241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923942144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078075.429250:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078075.429254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078075.429257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078075.429262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078075.429276:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078075.429284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078075/real 1769078075] req@ffff969e0a7ffb80 x1855007923942272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078075.429295:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923942272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078075.429310:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078075.429314:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078075.429317:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078075.429323:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078075.429341:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078075.429351:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078075/real 1769078075] req@ffff969e0a7fc000 x1855007923942400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078075.429362:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923942400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078075.429371:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078075.429374:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078075.429377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078075.748700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078076.452805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078076.772739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078077.476809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078077.796702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078078.500795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078078.820689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078079.524785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078079.844683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078080.548721:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078080.548736:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078080.548740:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078080.548753:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4769 00000100:00080000:3.0:1769078080.548765:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078080.548774:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769078080.548785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078080.548816:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923942528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078080.548843:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078080.548847:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078080.548850:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078080.548855:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4769 00000100:00080000:3.0:1769078080.548860:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078080.548865:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078080.548872:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923942528:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769078080.548877:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078080.548896:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923942656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078080.548904:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078080.548907:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078080.548909:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078080.548913:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4769 00000100:00080000:3.0:1769078080.548919:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078080.548923:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078080.548929:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769078080.548942:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078080.548966:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923942784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078080.548974:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078080.548976:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078080.548978:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078080.548984:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4769 00000100:00080000:3.0:1769078080.548988:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078080.548991:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078080.548993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078080/real 1769078080] req@ffff969e0a7fc380 x1855007923942528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078080.548996:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078080.549010:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923942912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078080.549019:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078080.549022:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078080.549027:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078080.549028:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923942528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078080.549033:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4769 00000100:00080000:3.0:1769078080.549036:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078080.549038:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078080.549042:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:0.0:1769078080.549056:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078080.549061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769078080.549065:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923943040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078080.549068:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923942528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078080.549107:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078080.549129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078080.549137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078080/real 1769078080] req@ffff969e0a7fc000 x1855007923942656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078080.549145:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923942656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078080.549150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078080.549152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078080.549155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078080.549161:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078080.549178:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078080.549189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078080/real 1769078080] req@ffff969e0a7ffb80 x1855007923942784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078080.549201:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923942784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078080.549208:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078080.549211:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078080.549213:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078080.549222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078080.549241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078080.549251:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078080/real 1769078080] req@ffff969e0a7fdc00 x1855007923942912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078080.549263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923942912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078080.549271:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078080.549274:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078080.549277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923942912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078080.549283:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078080.549301:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078080.549312:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078080/real 1769078080] req@ffff969e0a7ff800 x1855007923943040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078080.549322:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923943040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078080.549331:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078080.549334:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078080.549337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078080.868715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078081.572764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078081.892696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078082.596766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078082.916701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078083.620743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078083.940673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078084.644780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078084.964719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078085.668690:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078085.668701:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078085.668705:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078085.668721:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4774 00000100:00080000:3.0:1769078085.668732:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078085.668738:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:0.0:1769078085.668751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078085.668771:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923943168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078085.668791:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078085.668793:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078085.668794:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078085.668797:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4774 00000100:00080000:3.0:1769078085.668800:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078085.668804:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078085.668811:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078085.668815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923943168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078085.668821:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923943296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078085.668830:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078085.668832:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078085.668833:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078085.668835:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4774 00000100:00080000:3.0:1769078085.668837:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078085.668839:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078085.668853:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078085.668861:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923943424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078085.668868:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078085.668872:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078085.668873:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078085.668876:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078085.668879:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4774 00000100:00080000:3.0:1769078085.668883:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078085.668886:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078085.668890:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078085.668898:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923943552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078085.668905:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078085/real 1769078085] req@ffff969e0a7fd500 x1855007923943168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078085.668906:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078085.668913:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078085.668916:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078085.668922:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4774 00000100:00080000:3.0:1769078085.668925:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078085.668927:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078085.668931:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078085.668932:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923943168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078085.668940:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923943680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078085.668943:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078085.668951:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078085.668955:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923943168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078085.668975:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078085.668989:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078085.668996:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078085/real 1769078085] req@ffff969e0a7fd880 x1855007923943296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078085.669004:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923943296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078085.669009:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078085.669011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078085.669012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078085.669016:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078085.669025:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078085.669030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078085/real 1769078085] req@ffff969e0a7fd180 x1855007923943424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078085.669043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923943424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078085.669049:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078085.669050:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078085.669052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078085.669056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078085.669067:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078085.669072:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078085/real 1769078085] req@ffff969e0a7fe300 x1855007923943552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078085.669079:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923943552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078085.669084:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078085.669085:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078085.669087:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078085.669090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078085.669098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078085.669103:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078085/real 1769078085] req@ffff969e0a7fca80 x1855007923943680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078085.669109:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923943680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078085.669122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078085.669124:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078085.669126:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078085.988662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078086.692749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078087.012689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078087.716742:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078088.036649:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078088.740745:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078089.060713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078089.764808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078090.084711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078090.788708:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078090.788718:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078090.788722:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078090.788730:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4779 00010000:00010000:3.0:1769078090.788735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078090.788741:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078090.788749:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078090.788782:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923943808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078090.788804:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078090.788807:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078090.788808:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078090.788811:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4779 00000100:00080000:0.0:1769078090.788814:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078090.788816:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078090.788824:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078090.788834:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923943936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078090.788850:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923943808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078090.788875:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078090.788880:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078090.788882:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078090.788887:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4779 00000100:00080000:0.0:1769078090.788892:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078090.788897:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078090.788903:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078090.788917:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923944064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078090.788931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078090.788932:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078090.788938:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078090.788940:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078090.788945:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4779 00000100:00080000:0.0:1769078090.788949:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078090.788953:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078090.788958:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078090.788973:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078090/real 1769078090] req@ffff969e06a60380 x1855007923943808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078090.788981:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923944192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078090.788999:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769078090.789004:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923943808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078090.789005:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078090.789010:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078090.789019:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078090.789022:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4779 00000100:00080000:2.0:1769078090.789027:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078090.789035:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923943808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078090.789037:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078090.789047:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078090.789054:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078090.789070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078090.789071:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923944320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078090.789105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078090.789122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078090/real 1769078090] req@ffff969e06a61500 x1855007923943936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078090.789137:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923943936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078090.789159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078090.789163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078090.789168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923943936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078090.789175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078090.789199:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078090.789245:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078090/real 1769078090] req@ffff969e06a62d80 x1855007923944064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078090.789259:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923944064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078090.789268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078090.789272:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078090.789275:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078090.789282:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078090.789303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078090.789314:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078090/real 1769078090] req@ffff969e06a60700 x1855007923944192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078090.789327:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923944192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078090.789336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078090.789339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078090.789349:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078090.789366:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078090.789386:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078090.789398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078090/real 1769078090] req@ffff969e06a63800 x1855007923944320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078090.789411:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923944320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078090.789420:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078090.789424:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078090.789427:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078091.108703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078091.812797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078092.132712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078092.836738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078093.156700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078093.860795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078094.180700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078094.884856:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078095.204732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078095.908781:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078095.908790:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078095.908795:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078095.908804:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4785 00000100:00080000:0.0:1769078095.908816:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078095.908823:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078095.908860:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923944448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769078095.908864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078095.908887:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078095.908892:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078095.908894:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078095.908900:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4785 00000100:00100000:2.0:1769078095.908901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923944448:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078095.908906:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078095.908913:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078095.908924:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078095.908950:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923944576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078095.908959:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078095.908978:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078095.908979:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078095.908983:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4785 00000100:00080000:0.0:1769078095.908986:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078095.908990:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078095.908995:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078095.909006:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923944704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078095.909017:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078095.909020:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078095.909023:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078095.909027:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944576:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078095.909029:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4785 00000100:00080000:0.0:1769078095.909034:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078095.909038:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078095.909041:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078095.909059:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923944832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078095.909068:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078095.909072:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078095.909074:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769078095.909078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944704:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078095.909080:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4785 00000100:00080000:0.0:1769078095.909086:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078095.909090:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078095.909101:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078095.909111:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923944960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078095.909143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078095.909181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078096.228711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078096.932823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078097.252701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078097.956762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078098.276707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078098.980816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078098.980908:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769078098.980953:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078098.980972:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078098.981019:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078098.981043:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078098.981090:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078098.981103:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:0.0:1769078098.981132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078095/real 1769078098] req@ffff969e06a62d80 x1855007923944960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078098.981161:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923944960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078098.981174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078098.981180:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078098.981185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078098.981194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078095/real 1769078098] req@ffff969e06a60700 x1855007923944832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078098.981206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923944832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078098.981215:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078098.981219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078098.981223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078098.981230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078095/real 1769078098] req@ffff969e06a60a80 x1855007923944576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078098.981242:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923944576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078098.981265:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078098.981270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078098.981274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078098.981282:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078095/real 1769078098] req@ffff969e06a63800 x1855007923944704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078098.981296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923944704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078098.981304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078098.981308:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078098.981311:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923944704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078098.981319:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078095/real 1769078098] req@ffff969e06a60000 x1855007923944448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078098.981332:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923944448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078098.981340:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078098.981344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078098.981347:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923944448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078099.300690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078100.004755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078100.324666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078101.028709:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078101.028725:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078101.028729:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078101.028738:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4790 00000100:00080000:0.0:1769078101.028765:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078101.028774:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078101.028808:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923945088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078101.028808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078101.028844:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078101.028849:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078101.028851:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078101.028856:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4790 00000100:00080000:0.0:1769078101.028860:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078101.028865:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078101.028875:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078101.028880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923945088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078101.028887:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923945216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078101.028896:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078101.028900:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078101.028901:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078101.028905:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4790 00000100:00080000:0.0:1769078101.028914:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078101.028918:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078101.028923:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078101.028932:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923945344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078101.028941:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1769078101.028945:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078101.028951:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078101.028953:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078101.028957:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4790 00000100:00080000:0.0:1769078101.028961:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078101.028965:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078101.028970:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078101.028981:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923945472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078101.028986:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078101/real 1769078101] req@ffff969e06a60000 x1855007923945088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078101.028991:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078101.028995:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078101.028997:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078101.029001:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4790 00000100:00080000:0.0:1769078101.029004:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078101.029007:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078101.029010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923945088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078101.029012:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078101.029020:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923945600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078101.029021:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078101.029026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078101.029044:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923945088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078101.029071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078101.029095:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078101.029109:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078101/real 1769078101] req@ffff969e06a63800 x1855007923945216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078101.029121:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923945216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078101.029130:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078101.029133:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078101.029137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078101.029151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078101.029191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078101.029203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078101/real 1769078101] req@ffff969e06a60a80 x1855007923945344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078101.029215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923945344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078101.029223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078101.029226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078101.029230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078101.029236:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078101.029256:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078101.029266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078101/real 1769078101] req@ffff969e06a60700 x1855007923945472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078101.029278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923945472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078101.029287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078101.029291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078101.029295:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078101.029302:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078101.029339:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078101.029356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078101/real 1769078101] req@ffff969e06a62d80 x1855007923945600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078101.029383:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923945600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078101.029393:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078101.029397:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078101.029400:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078101.348685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078102.052787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078102.372727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078103.076748:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078103.396652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078104.100753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078104.420696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078105.124777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078105.444658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078106.148706:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078106.148719:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078106.148724:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078106.148736:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4795 00000100:00080000:2.0:1769078106.148749:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1769078106.148753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078106.148759:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078106.148795:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923945728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078106.148822:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078106.148824:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078106.148826:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078106.148829:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4795 00000100:00080000:2.0:1769078106.148832:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078106.148836:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078106.148845:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078106.148860:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923945856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078106.148875:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923945728:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769078106.148896:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078106.148899:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078106.148902:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078106.148918:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4795 00000100:00080000:2.0:1769078106.148924:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078106.148929:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078106.148935:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078106.148967:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923945984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078106.148974:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078106.148984:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078106.148989:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078106.148991:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078106.148997:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4795 00000100:00080000:2.0:1769078106.149001:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078106.149006:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078106.149012:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078106.149019:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078106/real 1769078106] req@ffff969e0a46d500 x1855007923945728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078106.149028:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923946112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078106.149046:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078106.149052:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923945728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078106.149052:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078106.149057:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078106.149066:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4795 00000100:00080000:2.0:1769078106.149091:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078106.149095:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078106.149099:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769078106.149105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:2.0:1769078106.149110:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078106.149114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923945728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078106.149128:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923946240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078106.149149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078106.149177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078106.149190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078106/real 1769078106] req@ffff969e0a46ea00 x1855007923945856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078106.149202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923945856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078106.149211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078106.149215:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078106.149219:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078106.149226:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078106.149247:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078106.149258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078106/real 1769078106] req@ffff969e0a46c380 x1855007923945984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078106.149270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923945984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078106.149279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078106.149283:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078106.149286:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923945984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078106.149300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078106.149324:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078106.149335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078106/real 1769078106] req@ffff969e0a46ce00 x1855007923946112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078106.149347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923946112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078106.149356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078106.149360:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078106.149363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078106.149369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078106.149390:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078106.149401:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078106/real 1769078106] req@ffff969e0a46f800 x1855007923946240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078106.149413:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923946240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078106.149422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078106.149425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078106.149429:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078106.468702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078107.172757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078107.492695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078108.196765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078108.516754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078109.220778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078109.540693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078110.244736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078110.564715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078111.268751:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078111.268762:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078111.268767:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078111.268779:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4800 00000100:00080000:2.0:1769078111.268792:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078111.268803:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078111.268873:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923946368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1769078111.268888:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078111.268911:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078111.268917:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078111.268919:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078111.268932:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4800 00000100:00080000:2.0:1769078111.268936:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078111.268941:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078111.268952:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078111.268961:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923946368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078111.268972:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923946496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078111.268982:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078111.269006:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078111.269008:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078111.269019:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4800 00000100:00080000:2.0:1769078111.269025:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078111.269028:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078111.269033:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078111.269045:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923946624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078111.269053:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078111.269057:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078111.269058:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078111.269063:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078111.269070:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4800 00000100:00080000:2.0:1769078111.269075:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078111.269078:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078111.269082:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078111.269093:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923946752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078111.269103:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078111.269106:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078111.269108:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078111.269111:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4800 00000100:00100000:1.0:1769078111.269117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078111/real 1769078111] req@ffff969e0a46d880 x1855007923946368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078111.269118:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078111.269129:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078111.269144:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078111.269151:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923946368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078111.269157:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923946880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078111.269181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078111.269196:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078111.269202:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923946368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078111.269230:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078111.269263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078111.269294:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078111/real 1769078111] req@ffff969e0a46f480 x1855007923946496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078111.269308:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923946496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078111.269317:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078111.269319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078111.269322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078111.269327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078111.269345:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078111.269353:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078111/real 1769078111] req@ffff969e0a46f800 x1855007923946624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078111.269361:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923946624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078111.269370:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078111.269373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078111.269377:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078111.269384:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078111.269406:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078111.269417:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078111/real 1769078111] req@ffff969e0a46ce00 x1855007923946752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078111.269429:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923946752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078111.269440:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078111.269443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078111.269446:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078111.269450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078111.269466:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078111.269473:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078111/real 1769078111] req@ffff969e0a46c380 x1855007923946880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078111.269480:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923946880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078111.269497:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078111.269500:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078111.269502:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923946880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078111.588740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078112.292820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078112.612699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078113.316810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078113.636743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078114.340864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078114.660709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078115.364867:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078115.684747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078116.388733:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078116.388741:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078116.388744:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078116.388753:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4805 00000100:00080000:0.0:1769078116.388763:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078116.388795:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769078116.388822:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078116.388825:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923947008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078116.388844:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078116.388846:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078116.388848:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078116.388851:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4805 00000100:00080000:0.0:1769078116.388853:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078116.388856:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078116.388863:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078116.388871:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923947136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078116.388879:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078116.388883:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078116.388884:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078116.388887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923947008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078116.388889:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4805 00000100:00080000:0.0:1769078116.388894:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078116.388899:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078116.388920:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078116.388934:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923947264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078116.388941:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078116.388944:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078116.388946:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078116.388948:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4805 00000100:00080000:0.0:1769078116.388951:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078116.388955:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769078116.388960:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769078116.388961:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078116.388969:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923947392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078116.388976:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078116.388978:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078116.388980:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078116.388982:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4805 00000100:00080000:0.0:1769078116.388984:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078116.388986:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078116.388989:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078116.388996:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923947520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078116.389000:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078116/real 1769078116] req@ffff969e06a61500 x1855007923947008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078116.389023:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923947008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078116.389033:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078116.389038:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078116.389041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923947008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078116.389071:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078116.389094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078116.389107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078116/real 1769078116] req@ffff969e06a60380 x1855007923947136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078116.389119:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923947136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078116.389127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078116.389131:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078116.389134:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078116.389140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078116.389157:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078116.389166:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078116/real 1769078116] req@ffff969e06a62680 x1855007923947264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078116.389178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923947264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078116.389185:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078116.389189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078116.389192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078116.389197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078116.389215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078116.389224:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078116/real 1769078116] req@ffff969e06a61f80 x1855007923947392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078116.389243:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923947392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078116.389251:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078116.389254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078116.389257:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078116.389262:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078116.389280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078116.389289:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078116/real 1769078116] req@ffff969e06a61180 x1855007923947520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078116.389300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923947520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078116.389308:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078116.389312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078116.389315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078116.708711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078117.412786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078117.732672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078118.436784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078118.756685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078119.460771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078119.780748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078120.484732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078120.804670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078121.508719:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078121.508727:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078121.508731:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078121.508739:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4810 00000100:00080000:3.0:1769078121.508750:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078121.508755:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078121.508774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078121.508798:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923947648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078121.508819:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078121.508821:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078121.508823:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078121.508825:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4810 00000100:00080000:3.0:1769078121.508828:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078121.508830:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078121.508837:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078121.508844:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923947776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078121.508851:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078121.508853:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078121.508854:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078121.508856:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4810 00000100:00080000:3.0:1769078121.508859:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078121.508862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923947648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078121.508863:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078121.508867:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078121.508874:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923947904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078121.508881:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078121.508883:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078121.508884:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078121.508890:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4810 00000100:00080000:3.0:1769078121.508893:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078121.508896:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078121.508899:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078121.508907:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923948032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078121.508913:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078121.508916:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078121.508916:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078121.508918:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4810 00000100:00080000:3.0:1769078121.508920:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078121.508922:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078121.508924:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078121.508926:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078121.508933:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923948160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078121.508962:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078121/real 1769078121] req@ffff969e0a7fdf80 x1855007923947648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078121.508980:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923947648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078121.509003:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078121.509008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078121.509012:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923947648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078121.509020:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078121.509039:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078121.509050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078121/real 1769078121] req@ffff969e0a7fea00 x1855007923947776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078121.509063:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923947776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078121.509072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078121.509075:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078121.509079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078121.509099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078121.509129:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078121.509138:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078121/real 1769078121] req@ffff969e0a7fc700 x1855007923947904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078121.509150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923947904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078121.509159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078121.509162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078121.509166:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923947904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078121.509177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078121.509194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078121.509203:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078121/real 1769078121] req@ffff969e0a7fce00 x1855007923948032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078121.509215:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923948032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078121.509223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078121.509226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078121.509230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078121.509235:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078121.509248:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078121.509257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078121/real 1769078121] req@ffff969e0a7fe680 x1855007923948160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078121.509267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923948160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078121.509275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078121.509278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078121.509281:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078121.828732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078122.532799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078122.852714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078123.557055:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078123.876742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078124.581166:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078124.900682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078125.604984:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078125.924729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078126.630178:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078126.630191:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078126.630205:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078126.630222:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4815 00000100:00080000:0.0:1769078126.630235:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078126.630243:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078126.630280:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923948288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078126.630318:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078126.630322:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078126.630334:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078126.630339:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4815 00000100:00080000:0.0:1769078126.630343:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078126.630347:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078126.630357:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078126.630371:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923948416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078126.630382:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078126.630385:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078126.630387:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078126.630391:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4815 00000100:00080000:0.0:1769078126.630394:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078126.630398:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078126.630403:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078126.630416:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923948544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078126.630441:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078126.630445:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078126.630447:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078126.630451:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4815 00000100:00080000:0.0:1769078126.630456:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078126.630460:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078126.630466:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078126.630478:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923948672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078126.630488:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078126.630492:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078126.630494:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078126.630498:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4815 00000100:00080000:0.0:1769078126.630501:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078126.630506:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078126.630511:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078126.630521:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923948800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078126.630635:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923948288:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078126.630704:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078126.630737:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078126/real 1769078126] req@ffff969e06a63100 x1855007923948288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078126.630764:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923948288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078126.630774:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078126.630778:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078126.630782:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923948288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078126.630790:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078126.630816:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078126.630829:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078126/real 1769078126] req@ffff969e06a61880 x1855007923948416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078126.630841:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923948416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078126.630850:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078126.630854:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078126.630857:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078126.630863:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078126.630886:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078126.630899:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078126/real 1769078126] req@ffff969e06a61180 x1855007923948544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078126.630911:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923948544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078126.630932:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078126.630936:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078126.630940:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078126.630946:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078126.630972:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078126.630984:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078126/real 1769078126] req@ffff969e06a61f80 x1855007923948672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078126.630998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923948672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078126.631007:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078126.631011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078126.631014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078126.631021:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078126.631044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078126.631055:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078126/real 1769078126] req@ffff969e06a62680 x1855007923948800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078126.631069:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923948800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078126.631078:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078126.631082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078126.631092:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923948800:192.168.204.143@tcp:8:kworker.0 00010000:00010000:3.0:1769078126.631215:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078126.948688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078127.652933:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078127.972768:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078128.676879:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078128.996736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078129.700689:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078130.020723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078130.724747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078131.044698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078131.748700:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078131.748710:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078131.748714:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078131.748723:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4820 00000100:00080000:0.0:1769078131.748753:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078131.748760:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078131.748792:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923948928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769078131.748803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078131.748816:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078131.748821:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078131.748822:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078131.748826:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4820 00000100:00080000:0.0:1769078131.748828:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078131.748831:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078131.748838:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078131.748848:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923949056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078131.748854:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923948928:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078131.748860:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078131.748863:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078131.748864:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078131.748867:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4820 00000100:00080000:0.0:1769078131.748869:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078131.748871:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078131.748875:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078131.748883:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923949184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078131.748890:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078131.748898:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078131.748901:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078131.748905:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4820 00000100:00080000:0.0:1769078131.748912:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078131.748916:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078131.748922:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078131.748931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078131.748944:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923949312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078131.748957:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078131.748961:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078131.748963:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078131.748968:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078131/real 1769078131] req@ffff969e06a60380 x1855007923948928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078131.748970:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4820 00000100:00080000:0.0:1769078131.748979:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078131.748985:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078131.748990:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078131.748991:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923948928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078131.749002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1769078131.749003:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923949440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078131.749008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078131.749019:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923948928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078131.749046:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078131.749070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078131.749081:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078131/real 1769078131] req@ffff969e06a61500 x1855007923949056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078131.749090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923949056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078131.749096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078131.749098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078131.749100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078131.749104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078131.749116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078131.749131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078131/real 1769078131] req@ffff969e06a62d80 x1855007923949184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078131.749139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923949184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078131.749145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078131.749147:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078131.749149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078131.749153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078131.749165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078131.749171:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078131/real 1769078131] req@ffff969e06a60700 x1855007923949312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078131.749178:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923949312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078131.749183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078131.749185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078131.749187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078131.749190:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078131.749200:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078131.749205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078131/real 1769078131] req@ffff969e06a60a80 x1855007923949440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078131.749212:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923949440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078131.749221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078131.749224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078131.749226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078132.068727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078132.772778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078133.092707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078133.796785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078134.116688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078134.820795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078135.140727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078135.844772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078136.164692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078136.868721:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078136.868738:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078136.868743:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078136.868758:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4826 00000100:00080000:3.0:1769078136.868769:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769078136.868772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078136.868776:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078136.868808:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923949568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078136.868829:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078136.868832:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078136.868834:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078136.868837:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4826 00000100:00080000:3.0:1769078136.868839:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078136.868857:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078136.868865:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078136.868881:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923949696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078136.868882:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923949568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078136.868895:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078136.868899:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078136.868901:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078136.868904:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4826 00000100:00080000:3.0:1769078136.868909:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078136.868913:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078136.868919:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078136.868932:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923949824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078136.868943:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078136.868947:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078136.868949:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078136.868953:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4826 00000100:00080000:3.0:1769078136.868959:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078136.868964:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078136.868968:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078136.868973:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078136.868980:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923949952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078136.868988:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078136.868995:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078136.868997:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078136.869000:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4826 00000100:00080000:3.0:1769078136.869003:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078136.869007:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078136.869011:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078136.869017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078136/real 1769078136] req@ffff969e0a7fed80 x1855007923949568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078136.869023:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923950080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078136.869042:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923949568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078136.869062:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078136.869068:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078136.869075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923949568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078136.869100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078136.869125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078136.869139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078136/real 1769078136] req@ffff969e0a7fe680 x1855007923949696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078136.869153:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923949696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078136.869163:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078136.869166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078136.869170:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078136.869185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078136.869206:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078136.869217:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078136/real 1769078136] req@ffff969e0a7fce00 x1855007923949824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078136.869230:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923949824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078136.869239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078136.869242:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078136.869246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078136.869252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078136.869269:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078136.869279:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078136/real 1769078136] req@ffff969e0a7fc700 x1855007923949952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078136.869291:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923949952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078136.869300:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078136.869303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078136.869306:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923949952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078136.869313:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078136.869332:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078136.869344:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078136/real 1769078136] req@ffff969e0a7fea00 x1855007923950080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078136.869365:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923950080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078136.869376:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078136.869379:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078136.869383:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078137.188718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078137.892795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078138.212759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078138.916785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078139.236697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078139.940790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078140.260738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078140.964756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078141.284708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078141.988742:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078141.988753:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078141.988758:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078141.988768:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4831 00000100:00080000:0.0:1769078141.988780:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078141.988786:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078141.988789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078141.988831:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923950208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078141.988875:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078141.988878:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078141.988880:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078141.988884:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4831 00000100:00080000:0.0:1769078141.988887:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078141.988889:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078141.988899:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078141.988912:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923950336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078141.988920:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078141.988924:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769078141.988925:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923950208:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078141.988926:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078141.988934:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4831 00000100:00080000:0.0:1769078141.988937:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078141.988939:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078141.988942:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078141.988951:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923950464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078141.988958:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078141.988960:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078141.988962:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078141.988965:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4831 00000100:00080000:0.0:1769078141.988984:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078141.988988:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078141.988992:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078141.988999:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078141.989003:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923950592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078141.989011:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078141.989020:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078141.989022:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078141.989025:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4831 00000100:00080000:0.0:1769078141.989027:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078141.989030:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078141.989033:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078141.989041:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923950720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078141.989057:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078141/real 1769078141] req@ffff969e06a63800 x1855007923950208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078141.989077:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923950208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078141.989086:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078141.989090:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078141.989093:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923950208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078141.989123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078141.989141:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078141.989150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078141/real 1769078141] req@ffff969e06a60000 x1855007923950336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078141.989158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923950336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078141.989174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078141.989177:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078141.989179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078141.989184:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078141.989197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078141.989204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078141/real 1769078141] req@ffff969e06a60a80 x1855007923950464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078141.989211:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923950464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078141.989217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078141.989219:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078141.989221:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078141.989224:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078141.989234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078141.989239:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078141/real 1769078141] req@ffff969e06a60700 x1855007923950592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078141.989246:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923950592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078141.989252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078141.989254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078141.989260:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078141.989264:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078141.989278:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078141.989287:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078141/real 1769078141] req@ffff969e06a62d80 x1855007923950720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078141.989299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923950720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078141.989309:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078141.989312:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078141.989315:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078142.308690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078143.012813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078143.332704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078144.036799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078144.356686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078145.060777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078145.380735:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078146.084779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078146.404693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078147.108722:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078147.108732:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078147.108736:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078147.108746:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4836 00010000:00010000:2.0:1769078147.108757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078147.108757:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078147.108764:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078147.108811:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923950848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078147.108834:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078147.108838:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078147.108840:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078147.108844:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4836 00000100:00080000:3.0:1769078147.108848:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078147.108854:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078147.108864:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078147.108880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923950848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078147.108882:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923950976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078147.108897:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078147.108900:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078147.108902:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078147.108904:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4836 00000100:00080000:3.0:1769078147.108908:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078147.108911:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078147.108915:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078147.108925:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923951104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078147.108931:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078147.108934:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078147.108935:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078147.108944:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4836 00000100:00080000:3.0:1769078147.108950:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078147.108953:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078147.108957:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078147.108963:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078147.108968:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923951232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078147.108976:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078147.108978:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078147.108980:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078147.108982:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4836 00000100:00080000:3.0:1769078147.108992:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078147.108995:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078147.108999:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078147.109011:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923951360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078147.109013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078147/real 1769078147] req@ffff969e0a7fdf80 x1855007923950848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078147.109042:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923950848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078147.109054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078147.109059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078147.109064:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923950848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078147.109090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078147.109114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078147.109137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078147/real 1769078147] req@ffff969e0a7fca80 x1855007923950976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078147.109150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923950976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078147.109159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078147.109163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078147.109167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923950976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078147.109173:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078147.109194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078147.109205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078147/real 1769078147] req@ffff969e0a7fe300 x1855007923951104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078147.109218:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923951104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078147.109227:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078147.109230:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078147.109234:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078147.109240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078147.109295:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078147.109307:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078147/real 1769078147] req@ffff969e0a7fd180 x1855007923951232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078147.109319:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923951232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078147.109335:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078147.109339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078147.109343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078147.109349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078147.109373:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078147.109385:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078147/real 1769078147] req@ffff969e0a7fd880 x1855007923951360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078147.109397:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923951360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078147.109406:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078147.109409:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078147.109412:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078147.428740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078148.132809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078148.452730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078149.156712:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078149.476710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078150.180764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078150.500737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078151.204775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078151.524758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078152.228721:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078152.228733:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078152.228738:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078152.228749:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4841 00000100:00080000:0.0:1769078152.228760:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078152.228767:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078152.228797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078152.228818:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923951488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078152.228840:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078152.228862:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078152.228864:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078152.228869:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4841 00000100:00080000:0.0:1769078152.228872:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078152.228876:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078152.228887:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078152.228899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923951488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078152.228906:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923951616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078152.228920:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078152.228924:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078152.228926:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078152.228929:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4841 00000100:00080000:0.0:1769078152.228933:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078152.228936:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078152.228940:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078152.228951:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923951744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078152.228958:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078152.228961:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078152.228963:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078152.228965:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4841 00000100:00080000:0.0:1769078152.228968:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078152.228970:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078152.228980:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078152.228987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078152.228991:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923951872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078152.229000:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078152.229003:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078152.229004:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078152.229007:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4841 00000100:00080000:0.0:1769078152.229011:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078152.229013:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078152.229017:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078152.229025:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923952000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078152.229037:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078152/real 1769078152] req@ffff969e06a61500 x1855007923951488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078152.229060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923951488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078152.229072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078152.229078:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078152.229083:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923951488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078152.229132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078152.229158:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078152.229186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078152/real 1769078152] req@ffff969e06a60380 x1855007923951616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078152.229207:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923951616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078152.229217:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078152.229221:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078152.229224:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078152.229232:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078152.229254:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078152.229266:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078152/real 1769078152] req@ffff969e06a62680 x1855007923951744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078152.229278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923951744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078152.229286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078152.229290:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078152.229293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078152.229300:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078152.229318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078152.229327:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078152/real 1769078152] req@ffff969e06a61f80 x1855007923951872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078152.229346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923951872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078152.229356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078152.229359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078152.229363:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923951872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078152.229369:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078152.229388:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078152.229398:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078152/real 1769078152] req@ffff969e06a61180 x1855007923952000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078152.229410:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923952000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078152.229425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078152.229429:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078152.229433:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078152.548739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078153.252786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078153.572741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078154.276829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078154.596713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078155.300802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078155.620701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078156.324826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078156.644717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078157.348708:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078157.348719:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078157.348724:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078157.348735:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4846 00000100:00080000:0.0:1769078157.348746:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078157.348753:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769078157.348780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078157.348802:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923952128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078157.348832:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078157.348837:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078157.348839:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078157.348844:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4846 00000100:00080000:0.0:1769078157.348862:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078157.348867:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078157.348878:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078157.348879:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923952128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078157.348896:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923952256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078157.348909:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078157.348914:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078157.348916:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078157.348921:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4846 00000100:00080000:0.0:1769078157.348935:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078157.348938:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078157.348942:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078157.348953:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078157.348954:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923952384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078157.348963:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078157.348966:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078157.348968:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078157.348971:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4846 00000100:00080000:0.0:1769078157.348974:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078157.348977:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078157.348980:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078157.348989:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923952512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078157.348997:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078157/real 1769078157] req@ffff969e06a61880 x1855007923952128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078157.349018:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078157.349023:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078157.349025:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078157.349029:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4846 00000100:00080000:0.0:1769078157.349033:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078157.349037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923952128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078157.349038:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078157.349044:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078157.349054:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923952640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078157.349075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078157.349096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078157.349102:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923952128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078157.349126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078157.349153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078157.349167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078157/real 1769078157] req@ffff969e06a63100 x1855007923952256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078157.349181:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923952256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078157.349190:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078157.349195:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078157.349198:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078157.349205:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078157.349224:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078157.349235:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078157/real 1769078157] req@ffff969e06a61180 x1855007923952384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078157.349248:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923952384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078157.349257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078157.349261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078157.349264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078157.349277:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078157.349299:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078157.349310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078157/real 1769078157] req@ffff969e06a61f80 x1855007923952512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078157.349323:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923952512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078157.349332:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078157.349336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078157.349339:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078157.349346:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078157.349365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078157.349376:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078157/real 1769078157] req@ffff969e06a62680 x1855007923952640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078157.349388:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923952640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078157.349397:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078157.349400:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078157.349403:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078157.668697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078158.372818:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078158.692699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078159.396777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078159.716696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078160.420761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078160.740696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078161.444781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078161.764688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078162.468731:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078162.468743:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078162.468749:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078162.468762:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4851 00000100:00080000:3.0:1769078162.468774:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769078162.468781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078162.468784:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078162.468823:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923952768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078162.468853:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078162.468857:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078162.468859:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078162.468865:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4851 00000100:00080000:3.0:1769078162.468869:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078162.468875:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078162.468886:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078162.468896:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923952768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078162.468919:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923952896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078162.468932:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078162.468936:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078162.468938:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078162.468943:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4851 00000100:00080000:3.0:1769078162.468948:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078162.468952:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078162.468958:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078162.468971:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923953024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078162.468982:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078162.468986:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078162.468988:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078162.469011:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4851 00000100:00080000:3.0:1769078162.469016:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078162.469024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952896:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769078162.469024:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078162.469032:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078162.469049:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923953152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078162.469062:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078162.469066:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078162.469068:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078162.469077:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953024:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769078162.469083:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4851 00000100:00080000:3.0:1769078162.469091:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078162.469095:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078162.469103:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078162.469114:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078162.469120:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923953280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078162.469151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078162.788734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078163.492786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078163.812705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078164.516812:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078164.836731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078165.540839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1769078165.541026:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:3.0:1769078165.541090:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769078165.541116:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769078165.541167:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769078165.541179:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769078165.541189:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769078165.541201:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769078165.541439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078162/real 1769078165] req@ffff969e0a7fc000 x1855007923953280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.541474:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923953280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.541492:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.541499:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078165.541505:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078165.541515:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078162/real 1769078165] req@ffff969e0a7ffb80 x1855007923953152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.541529:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923953152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.541538:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.541541:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078165.541545:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078165.541554:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078162/real 1769078165] req@ffff969e0a7ff800 x1855007923952896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.541567:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923952896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.541638:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.541644:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078165.541648:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923952896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078165.541657:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078162/real 1769078165] req@ffff969e0a7fdc00 x1855007923953024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.541672:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923953024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.541681:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.541686:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078165.541690:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078165.541698:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078162/real 1769078165] req@ffff969e0a7fd500 x1855007923952768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078165.541712:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923952768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078165.541721:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078165.541725:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078165.541729:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923952768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078165.860659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078166.564805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078166.884684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078167.588782:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078167.588791:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078167.588796:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769078167.588797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078167.588805:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4856 00000100:00080000:1.0:1769078167.588817:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078167.588823:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078167.588851:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923953408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078167.588875:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078167.588879:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078167.588882:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078167.588887:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4856 00000100:00080000:1.0:1769078167.588890:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078167.588895:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078167.588904:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078167.588934:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923953536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078167.588947:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078167.588950:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078167.588953:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078167.588960:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4856 00000100:00080000:1.0:1769078167.588966:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078167.588974:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923953408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769078167.588977:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078167.588988:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078167.589002:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923953664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078167.589013:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078167.589018:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078167.589020:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078167.589024:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4856 00000100:00080000:1.0:1769078167.589039:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078167.589044:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078167.589050:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769078167.589058:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078167.589065:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923953792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078167.589078:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078167.589082:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078167.589084:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078167.589088:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4856 00000100:00080000:1.0:1769078167.589094:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078167.589101:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078167.589102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078167/real 1769078167] req@ffff969e10b62680 x1855007923953408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078167.589108:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078167.589127:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923953920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078167.589135:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923953408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078167.589168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078167.589174:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078167.589179:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923953408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078167.589188:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078167.589215:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078167.589230:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078167/real 1769078167] req@ffff969e10b62a00 x1855007923953536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078167.589274:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923953536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078167.589286:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078167.589289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078167.589293:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078167.589322:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078167.589346:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078167.589360:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078167/real 1769078167] req@ffff969e10b60a80 x1855007923953664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078167.589373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923953664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078167.589382:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078167.589386:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078167.589389:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078167.589398:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078167.589424:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078167.589434:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078167/real 1769078167] req@ffff969e10b61f80 x1855007923953792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078167.589446:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923953792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078167.589468:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078167.589472:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078167.589476:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078167.589483:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078167.589510:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078167.589520:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078167/real 1769078167] req@ffff969e10b61500 x1855007923953920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078167.589532:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923953920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078167.589541:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078167.589544:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078167.589547:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923953920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078167.908703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078168.612928:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078168.932722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078169.636769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078169.956711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078170.660777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078170.980690:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078171.684746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078172.004756:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078172.708676:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078172.708685:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078172.708689:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078172.708698:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4861 00000100:00080000:1.0:1769078172.708708:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078172.708714:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078172.708748:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923954048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078172.708777:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078172.708781:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078172.708783:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078172.708788:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4861 00000100:00080000:1.0:1769078172.708792:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078172.708798:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078172.708807:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078172.708837:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923954176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078172.708847:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078172.708850:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078172.708852:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078172.708859:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923954048:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769078172.708859:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4861 00000100:00080000:1.0:1769078172.708864:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078172.708872:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078172.708878:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078172.708899:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923954304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078172.708911:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078172.708916:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078172.708918:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078172.708921:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4861 00000100:00080000:1.0:1769078172.708925:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078172.708930:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769078172.708933:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078172.708939:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078172.708950:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923954432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078172.708957:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078172.708960:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078172.708961:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078172.708966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078172/real 1769078172] req@ffff969e10b61500 x1855007923954048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078172.708966:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4861 00000100:00080000:1.0:1769078172.708968:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078172.708975:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078172.708981:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078172.708991:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923954560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078172.708993:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923954048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078172.709016:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078172.709020:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078172.709023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923954048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078172.709076:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954176:192.168.204.143@tcp:38:kworker.0 00010000:00010000:3.0:1769078172.709096:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:0.0:1769078172.709105:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078172.709118:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078172/real 1769078172] req@ffff969e10b61f80 x1855007923954176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078172.709128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923954176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078172.709134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078172.709137:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078172.709139:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078172.709143:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078172.709156:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078172.709162:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078172/real 1769078172] req@ffff969e10b60a80 x1855007923954304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078172.709169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923954304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078172.709175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078172.709176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078172.709178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078172.709182:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078172.709195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078172.709201:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078172/real 1769078172] req@ffff969e10b62a00 x1855007923954432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078172.709214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923954432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078172.709221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078172.709223:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078172.709225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078172.709228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078172.709241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078172.709247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078172/real 1769078172] req@ffff969e10b62680 x1855007923954560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078172.709254:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923954560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078172.709260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078172.709262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078172.709265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078173.028706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078173.732750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078174.052723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078174.756771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078175.076705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078175.780791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078176.100748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078176.804792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078177.124718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078177.828713:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078177.828726:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078177.828759:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078177.828769:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4867 00000100:00080000:1.0:1769078177.828780:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078177.828786:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078177.828817:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923954688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078177.828849:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078177.828854:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078177.828856:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078177.828859:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4867 00000100:00080000:1.0:1769078177.828862:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078177.828865:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078177.828872:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078177.828880:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923954816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078177.828888:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078177.828890:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078177.828892:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078177.828894:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4867 00000100:00080000:1.0:1769078177.828897:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078177.828899:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078177.828902:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078177.828911:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923954688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078177.828911:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923954944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078177.828928:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078177.828930:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078177.828931:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078177.828933:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4867 00000100:00080000:1.0:1769078177.828937:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078177.828940:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078177.828944:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078177.828952:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923955072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078177.828958:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078177.828961:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078177.828962:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078177.828964:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4867 00000100:00080000:1.0:1769078177.828984:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078177.828987:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078177.828990:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078177.828997:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923955200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078177.829004:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078177.829049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078177/real 1769078177] req@ffff969e10b60700 x1855007923954688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078177.829071:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923954688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1769078177.829090:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078177.829105:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078177.829110:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078177.829125:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923954688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078177.829133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078177.829160:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078177.829173:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078177/real 1769078177] req@ffff969e10b62d80 x1855007923954816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078177.829185:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923954816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078177.829193:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078177.829197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078177.829200:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078177.829225:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078177.829245:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078177.829256:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078177/real 1769078177] req@ffff969e10b60000 x1855007923954944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078177.829267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923954944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078177.829275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078177.829279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078177.829283:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923954944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078177.829295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078177.829317:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078177.829328:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078177/real 1769078177] req@ffff969e10b61180 x1855007923955072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078177.829341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923955072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078177.829350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078177.829353:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078177.829356:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078177.829362:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078177.829381:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078177.829391:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078177/real 1769078177] req@ffff969e10b60380 x1855007923955200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078177.829413:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923955200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078177.829422:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078177.829425:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078177.829428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078178.148699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078178.852831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078179.172705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078179.876805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078180.196713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078180.900773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078181.220691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078181.924790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078182.244743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078182.948738:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078182.948749:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078182.948753:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1769078182.948770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078182.948786:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4872 00000100:00080000:0.0:1769078182.948795:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078182.948801:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078182.948836:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923955328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078182.948862:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078182.948865:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078182.948866:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078182.948869:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4872 00000100:00080000:0.0:1769078182.948872:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078182.948876:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078182.948884:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078182.948895:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923955456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078182.948906:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078182.948909:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923955328:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078182.948910:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078182.948912:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078182.948926:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4872 00000100:00080000:0.0:1769078182.948929:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078182.948932:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078182.948934:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078182.948945:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923955584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078182.948960:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078182.948962:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078182.948963:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078182.948966:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4872 00000100:00080000:0.0:1769078182.948971:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078182.948974:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078182.948979:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078182.948984:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078182.948989:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923955712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078182.948996:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078182.949000:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078182.949002:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078182.949006:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4872 00000100:00080000:0.0:1769078182.949011:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078182.949019:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078182.949021:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078182/real 1769078182] req@ffff969e06a60380 x1855007923955328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078182.949031:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078182.949053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923955328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078182.949054:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923955840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078182.949123:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078182.949130:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078182.949135:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923955328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078182.949157:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078182.949172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078182.949180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078182/real 1769078182] req@ffff969e06a61500 x1855007923955456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078182.949191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923955456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078182.949200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078182.949204:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078182.949207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078182.949213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078182.949232:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078182.949243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078182/real 1769078182] req@ffff969e06a62d80 x1855007923955584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078182.949256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923955584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078182.949264:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078182.949268:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078182.949271:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078182.949278:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078182.949296:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078182.949303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078182/real 1769078182] req@ffff969e06a60700 x1855007923955712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078182.949315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923955712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078182.949320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078182.949322:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078182.949323:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078182.949327:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078182.949337:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078182.949345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078182/real 1769078182] req@ffff969e06a60a80 x1855007923955840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078182.949356:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923955840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078182.949364:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078182.949367:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078182.949370:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923955840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078183.268672:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078183.972767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078184.292734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078184.996794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078185.316719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078186.020718:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078186.340716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078187.044791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078187.364755:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078188.068695:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078188.068707:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078188.068713:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769078188.068714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078188.068724:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4877 00000100:00080000:3.0:1769078188.068736:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078188.068743:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078188.068786:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923955968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078188.068811:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078188.068819:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078188.068821:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078188.068826:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4877 00000100:00080000:3.0:1769078188.068830:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078188.068834:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078188.068842:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923955968:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769078188.068845:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078188.068863:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923956096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078188.068872:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078188.068875:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078188.068877:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078188.068881:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4877 00000100:00080000:3.0:1769078188.068884:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078188.068888:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078188.068908:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078188.068921:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923956224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078188.068927:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078188.068936:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078188.068940:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078188.068942:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078188.068946:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4877 00000100:00080000:3.0:1769078188.068950:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078188.068955:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078188.068960:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078188.068970:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078188/real 1769078188] req@ffff969e0a7fc380 x1855007923955968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078188.068973:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923956352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078188.068989:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078188.068993:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923955968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078188.068995:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078188.068998:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078188.069007:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4877 00000100:00080000:3.0:1769078188.069013:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078188.069017:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078188.069017:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769078188.069024:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:3.0:1769078188.069025:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078188.069029:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923955968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078188.069039:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923956480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078188.069057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078188.069073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078188.069081:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078188/real 1769078188] req@ffff969e0a7fd500 x1855007923956096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078188.069088:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923956096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078188.069094:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078188.069096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078188.069099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078188.069105:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078188.069121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078188.069131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078188/real 1769078188] req@ffff969e0a7fdc00 x1855007923956224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078188.069150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923956224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078188.069159:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078188.069162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078188.069165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078188.069171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078188.069196:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078188.069206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078188/real 1769078188] req@ffff969e0a7ff800 x1855007923956352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078188.069216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923956352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078188.069221:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078188.069223:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078188.069225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078188.069228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078188.069241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078188.069247:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078188/real 1769078188] req@ffff969e0a7ffb80 x1855007923956480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078188.069253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923956480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078188.069258:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078188.069259:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078188.069260:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078188.388754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078189.092793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078189.412723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078190.118523:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078190.436734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078191.140876:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078191.460730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078192.165117:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078192.484723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078193.189432:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078193.189451:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078193.189458:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078193.189473:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4882 00000100:00080000:1.0:1769078193.189525:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078193.189535:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078193.189586:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60380 x1855007923956608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078193.189634:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078193.189638:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078193.189640:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078193.189646:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4882 00000100:00080000:1.0:1769078193.189649:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078193.189653:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078193.189661:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078193.189671:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61180 x1855007923956736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078193.189677:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078193.189680:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078193.189681:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078193.189683:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4882 00000100:00080000:1.0:1769078193.189685:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078193.189687:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078193.189690:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078193.189696:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60000 x1855007923956864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078193.189711:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078193.189714:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078193.189715:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078193.189718:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4882 00000100:00080000:1.0:1769078193.189721:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078193.189725:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078193.189729:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078193.189738:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62d80 x1855007923956992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078193.189746:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078193.189749:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078193.189750:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078193.189754:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4882 00000100:00080000:1.0:1769078193.189756:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078193.189759:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078193.189763:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078193.189770:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60700 x1855007923957120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078193.189860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078193.189982:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923956608:192.168.204.143@tcp:250:kworker.0 00000800:00000100:0.0:1769078193.190042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078193.190075:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078193/real 1769078193] req@ffff969e10b60380 x1855007923956608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078193.190099:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60380 x1855007923956608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078193.190106:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078193.190109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078193.190112:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923956608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078193.190117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078193.190132:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078193.190139:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078193/real 1769078193] req@ffff969e10b61180 x1855007923956736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078193.190146:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61180 x1855007923956736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078193.190152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078193.190154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078193.190156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078193.190167:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078193.190180:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078193.190188:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078193/real 1769078193] req@ffff969e10b60000 x1855007923956864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078193.190200:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60000 x1855007923956864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078193.190216:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078193.190220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078193.190222:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078193.190228:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078193.190248:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078193.190258:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078193/real 1769078193] req@ffff969e10b62d80 x1855007923956992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078193.190267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62d80 x1855007923956992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078193.190276:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078193.190279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078193.190282:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923956992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078193.190288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078193.190303:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078193.190310:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078193/real 1769078193] req@ffff969e10b60700 x1855007923957120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078193.190317:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60700 x1855007923957120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078193.190322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078193.190324:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078193.190331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078193.508701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078194.212772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078194.532686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078195.237947:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078195.556709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078196.260774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078196.580692:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078197.284757:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078197.604686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078198.308712:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078198.308721:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078198.308725:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078198.308732:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4887 00000100:00080000:3.0:1769078198.308742:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078198.308748:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078198.308771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078198.308777:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923957248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078198.308799:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078198.308801:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078198.308802:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078198.308805:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4887 00000100:00080000:3.0:1769078198.308807:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078198.308809:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078198.308815:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078198.308824:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923957376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078198.308831:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078198.308839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923957248:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078198.308848:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078198.308850:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078198.308854:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4887 00000100:00080000:3.0:1769078198.308857:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078198.308860:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078198.308863:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078198.308873:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923957504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078198.308879:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078198.308882:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078198.308883:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078198.308885:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4887 00000100:00080000:3.0:1769078198.308887:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078198.308889:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078198.308891:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769078198.308897:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078198.308900:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923957632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078198.308906:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078198.308909:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078198.308910:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078198.308913:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4887 00000100:00080000:3.0:1769078198.308917:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078198.308920:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078198.308927:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078198.308930:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078198/real 1769078198] req@ffff969e0a7fc000 x1855007923957248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078198.308937:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923957760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078198.308948:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923957248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078198.308962:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078198.308965:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078198.308968:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923957248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078198.308983:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078198.308997:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078198.309003:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078198/real 1769078198] req@ffff969e0a7fd880 x1855007923957376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078198.309010:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923957376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078198.309015:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078198.309017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078198.309019:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078198.309022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078198.309031:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078198.309044:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078198/real 1769078198] req@ffff969e0a7fd180 x1855007923957504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078198.309051:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923957504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078198.309055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078198.309057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078198.309059:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078198.309062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078198.309073:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078198.309077:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078198/real 1769078198] req@ffff969e0a7fe300 x1855007923957632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078198.309083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923957632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078198.309088:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078198.309089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078198.309091:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078198.309094:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078198.309102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078198.309106:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078198/real 1769078198] req@ffff969e0a7fca80 x1855007923957760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078198.309112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923957760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078198.309126:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078198.309128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078198.309130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923957760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078198.628678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078199.332743:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078199.652714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078200.356833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078200.676674:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078201.380741:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078201.700698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078202.404769:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078202.724666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078203.428683:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078203.428693:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078203.428696:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078203.428706:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4892 00000100:00080000:3.0:1769078203.428720:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078203.428728:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078203.428752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078203.428768:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923957888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078203.428799:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078203.428803:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078203.428805:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078203.428809:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4892 00000100:00080000:3.0:1769078203.428818:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078203.428823:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078203.428832:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078203.428845:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923958016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078203.428849:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923957888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078203.428873:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078203.428875:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078203.428876:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078203.428880:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4892 00000100:00080000:3.0:1769078203.428883:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078203.428886:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078203.428890:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078203.428899:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923958144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078203.428907:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078203.428909:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078203.428910:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1769078203.428915:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078203.428915:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4892 00000100:00080000:3.0:1769078203.428921:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078203.428923:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078203.428927:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078203.428936:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923958272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078203.428942:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078203.428944:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078203.428946:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078203.428948:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4892 00000100:00100000:0.0:1769078203.428954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078203/real 1769078203] req@ffff969e0a7fdf80 x1855007923957888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078203.428955:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078203.428962:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078203.428974:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078203.428977:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923957888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078203.428984:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923958400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078203.428998:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078203.429004:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078203.429008:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923957888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078203.429031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078203.429051:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078203.429069:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078203/real 1769078203] req@ffff969e0a7fea00 x1855007923958016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078203.429081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923958016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078203.429089:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078203.429093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078203.429099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078203.429111:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078203.429128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078203.429136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078203/real 1769078203] req@ffff969e0a7fc700 x1855007923958144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078203.429147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923958144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078203.429155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078203.429158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078203.429161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078203.429166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078203.429179:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078203.429186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078203/real 1769078203] req@ffff969e0a7fce00 x1855007923958272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078203.429197:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923958272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078203.429205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078203.429207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078203.429210:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078203.429215:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078203.429227:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078203.429234:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078203/real 1769078203] req@ffff969e0a7fe680 x1855007923958400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078203.429250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923958400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078203.429255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078203.429257:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078203.429259:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078203.748712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078204.452756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078204.772645:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078205.476721:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078205.796707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078206.500747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078206.820691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078207.524794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078207.844719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078208.548702:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078208.548711:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078208.548715:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078208.548722:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4897 00000100:00080000:3.0:1769078208.548732:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078208.548737:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078208.548768:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923958528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078208.548774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078208.548791:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078208.548794:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078208.548795:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078208.548798:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4897 00000100:00080000:3.0:1769078208.548801:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078208.548803:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078208.548809:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078208.548835:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923958656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078208.548842:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078208.548844:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923958528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078208.548845:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078208.548848:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078208.548855:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4897 00000100:00080000:3.0:1769078208.548858:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078208.548860:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078208.548863:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078208.548871:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923958784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078208.548876:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078208.548878:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078208.548879:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078208.548882:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4897 00000100:00080000:3.0:1769078208.548883:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078208.548885:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078208.548888:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078208.548894:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923958912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078208.548899:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078208.548900:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078208.548901:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078208.548908:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4897 00000100:00080000:3.0:1769078208.548913:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078208.548917:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078208.548920:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078208.548926:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078208.548939:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923959040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078208.548953:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078208/real 1769078208] req@ffff969e0a7fed80 x1855007923958528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078208.548969:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923958528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078208.548976:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078208.548979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078208.548982:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923958528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078208.548999:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078208.549014:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078208.549024:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078208/real 1769078208] req@ffff969e0a7fe680 x1855007923958656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078208.549035:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923958656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078208.549055:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078208.549059:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078208.549063:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078208.549068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078208.549084:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078208.549091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078208/real 1769078208] req@ffff969e0a7fce00 x1855007923958784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078208.549097:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923958784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078208.549110:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078208.549112:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078208.549114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078208.549120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078208.549131:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078208.549137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078208/real 1769078208] req@ffff969e0a7fc700 x1855007923958912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078208.549143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923958912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078208.549148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078208.549150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078208.549156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923958912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078208.549159:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078208.549171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078208.549176:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078208/real 1769078208] req@ffff969e0a7fea00 x1855007923959040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078208.549183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923959040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078208.549187:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078208.549189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078208.549190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078208.868703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078209.572767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078209.892739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078210.596782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078210.916712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078211.620767:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078211.940697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078212.644775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078212.964713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078213.668711:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078213.668723:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078213.668728:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078213.668740:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4902 00000100:00080000:3.0:1769078213.668752:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078213.668760:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078213.668790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078213.668806:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923959168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078213.668842:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078213.668846:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078213.668848:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078213.668852:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4902 00000100:00080000:3.0:1769078213.668856:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078213.668861:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078213.668880:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923959168:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769078213.668889:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078213.668906:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923959296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078213.668918:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078213.668921:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078213.668923:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078213.668927:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4902 00000100:00080000:3.0:1769078213.668932:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078213.668936:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078213.668941:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769078213.668947:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078213.668954:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923959424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078213.668965:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078213.668969:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078213.668971:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078213.668975:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4902 00000100:00080000:3.0:1769078213.668980:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078213.668982:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078213/real 1769078213] req@ffff969e0a7fdf80 x1855007923959168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078213.668987:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078213.668995:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078213.668998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923959168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078213.669006:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078213.669011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769078213.669012:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923959552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078213.669015:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923959168:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078213.669029:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078213.669033:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078213.669040:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078213.669041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959296:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769078213.669051:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4902 00000100:00080000:3.0:1769078213.669057:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078213.669061:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078213.669063:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078213.669069:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078213.669080:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078213/real 1769078213] req@ffff969e0a7fca80 x1855007923959296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078213.669087:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923959680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078213.669092:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923959296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078213.669112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078213.669115:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078213.669118:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078213.669123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078213.669152:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078213.669160:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078213/real 1769078213] req@ffff969e0a7fe300 x1855007923959424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078213.669168:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923959424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078213.669173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078213.669175:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078213.669177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078213.669181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078213.669191:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078213.669196:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078213/real 1769078213] req@ffff969e0a7fd180 x1855007923959552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078213.669202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923959552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078213.669212:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078213.669214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078213.669216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078213.669227:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078213.669239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078213.669244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078213/real 1769078213] req@ffff969e0a7fd880 x1855007923959680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078213.669251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923959680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078213.669256:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078213.669258:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078213.669260:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078213.988716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078214.692763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078215.012686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078215.716701:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078216.036699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078216.740752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078217.060710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078217.764795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078218.084683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078218.788703:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078218.788713:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078218.788718:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078218.788727:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4907 00000100:00080000:0.0:1769078218.788738:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078218.788745:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078218.788785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078218.788790:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923959808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078218.788818:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078218.788833:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078218.788835:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078218.788840:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4907 00000100:00080000:0.0:1769078218.788844:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078218.788848:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923959808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078218.788851:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078218.788864:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078218.788878:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923959936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078218.788889:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078218.788892:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078218.788893:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078218.788897:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4907 00000100:00080000:0.0:1769078218.788902:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078218.788906:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078218.788910:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078218.788917:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078218.788922:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923960064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078218.788931:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078218.788935:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078218.788937:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078218.788940:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4907 00000100:00080000:0.0:1769078218.788945:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078218.788950:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078218.788953:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078218/real 1769078218] req@ffff969e06a60000 x1855007923959808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078218.788962:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078218.788977:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923960192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078218.788987:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078218.788987:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923959808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078218.788995:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078218.788998:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078218.789002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078218.789003:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4907 00000100:00080000:1.0:1769078218.789008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769078218.789009:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078218.789013:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923959808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078218.789017:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078218.789030:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078218.789039:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078218.789042:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923960320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078218.789077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078218.789091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078218/real 1769078218] req@ffff969e06a63800 x1855007923959936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078218.789102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923959936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078218.789107:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078218.789109:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078218.789111:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923959936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078218.789115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078218.789128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078218.789134:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078218/real 1769078218] req@ffff969e06a60a80 x1855007923960064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078218.789141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923960064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078218.789149:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078218.789151:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078218.789153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078218.789156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078218.789165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078218.789170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078218/real 1769078218] req@ffff969e06a60700 x1855007923960192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078218.789176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923960192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078218.789182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078218.789183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078218.789185:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078218.789194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078218.789203:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078218.789208:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078218/real 1769078218] req@ffff969e06a62d80 x1855007923960320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078218.789214:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923960320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078218.789219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078218.789220:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078218.789223:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078219.108686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078219.812779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078220.132664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078220.836738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078221.156715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078221.860771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078222.180675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078222.884746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078223.204717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078223.908738:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078223.908748:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078223.908753:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078223.908762:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4913 00000100:00080000:0.0:1769078223.908773:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078223.908779:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769078223.908796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078223.908813:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923960448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078223.908835:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078223.908837:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078223.908838:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078223.908845:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4913 00000100:00080000:0.0:1769078223.908860:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078223.908864:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078223.908873:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078223.908882:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923960448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078223.908887:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923960576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078223.908904:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078223.908906:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078223.908909:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078223.908912:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4913 00000100:00080000:0.0:1769078223.908914:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078223.908916:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078223.908920:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078223.908928:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923960704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078223.908934:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078223.908937:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078223.908938:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078223.908941:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4913 00000100:00080000:0.0:1769078223.908943:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078223.908946:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078223.908949:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078223.908955:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923960832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078223.908980:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078223.908982:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078223.908983:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078223.908986:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4913 00000100:00080000:0.0:1769078223.908990:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078223.908993:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078223.908997:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078223.909005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078223.909010:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923960960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078223.909049:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078223/real 1769078223] req@ffff969e06a61500 x1855007923960448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078223.909064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923960448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078223.909072:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078223.909076:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078223.909079:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923960448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078223.909103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078223.909121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078223.909132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078223/real 1769078223] req@ffff969e06a60380 x1855007923960576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078223.909144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923960576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078223.909153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078223.909156:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078223.909159:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078223.909166:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078223.909183:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078223.909192:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078223/real 1769078223] req@ffff969e06a62680 x1855007923960704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078223.909199:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923960704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078223.909205:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078223.909207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078223.909209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078223.909213:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078223.909222:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078223.909228:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078223/real 1769078223] req@ffff969e06a61f80 x1855007923960832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078223.909234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923960832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078223.909239:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078223.909241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078223.909242:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078223.909246:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078223.909255:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078223.909260:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078223/real 1769078223] req@ffff969e06a61180 x1855007923960960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078223.909266:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923960960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078223.909273:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078223.909275:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078223.909277:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923960960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078224.228691:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078224.932738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078225.252700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078225.956888:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078226.276710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078226.980686:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078227.300684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078228.004859:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078228.324696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078229.028745:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078229.028758:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078229.028763:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078229.028776:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4918 00000100:00080000:3.0:1769078229.028810:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078229.028817:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078229.028865:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923961088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078229.028898:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078229.028912:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078229.028917:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078229.028918:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078229.028922:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4918 00000100:00080000:3.0:1769078229.028926:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078229.028930:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078229.028940:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078229.028961:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923961216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078229.028962:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923961088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078229.028991:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078229.028995:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078229.028997:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078229.029003:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4918 00000100:00080000:3.0:1769078229.029006:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078229.029009:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078229.029013:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078229.029023:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923961344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078229.029032:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078229.029040:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078229.029041:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078229.029044:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4918 00000100:00080000:3.0:1769078229.029048:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078229.029051:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078229.029056:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078229.029070:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923961472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078229.029082:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078229.029087:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078229.029091:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078229.029097:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4918 00000100:00080000:3.0:1769078229.029102:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078229.029108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961216:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769078229.029115:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078229.029127:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078229.029147:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923961600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078229.029158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078229.029190:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078229.029220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078229.348733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078230.052809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078230.372727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078231.076781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078231.396733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078232.100787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769078232.100901:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769078232.100939:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769078232.100959:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078232.101001:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078232.101012:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078232.101024:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078232.101036:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769078232.101036:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078229/real 1769078232] req@ffff969e0a7ffb80 x1855007923961216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078232.101065:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923961216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078232.101079:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078232.101085:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078232.101088:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078232.101095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078229/real 1769078232] req@ffff969e0a7ff800 x1855007923961344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078232.101107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923961344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078232.101122:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078232.101126:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078232.101130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078232.101136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078229/real 1769078232] req@ffff969e0a7fdc00 x1855007923961472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078232.101150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923961472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078232.101157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078232.101160:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078232.101163:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078232.101170:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078229/real 1769078232] req@ffff969e0a7fc000 x1855007923961088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078232.101182:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923961088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078232.101188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078232.101191:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078232.101194:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923961088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078232.101220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078229/real 1769078232] req@ffff969e0a7fd500 x1855007923961600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078232.101231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923961600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078232.101240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078232.101243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078232.101246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078232.420659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078233.124770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078233.444688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078234.148750:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078234.148767:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1769078234.148814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078234.148814:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078234.148832:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4923 00000100:00080000:3.0:1769078234.148846:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078234.148855:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078234.148901:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923961728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078234.148935:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078234.148939:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078234.148941:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078234.148946:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4923 00000100:00080000:3.0:1769078234.148950:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078234.148955:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078234.148968:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078234.148985:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923961728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078234.148985:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923961856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078234.148997:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078234.149004:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078234.149006:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078234.149009:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4923 00000100:00080000:3.0:1769078234.149012:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078234.149016:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078234.149021:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078234.149033:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923961984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078234.149056:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078234.149059:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078234.149061:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078234.149066:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4923 00000100:00080000:3.0:1769078234.149071:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078234.149074:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769078234.149080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769078234.149080:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078234.149096:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923962112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078234.149105:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078234.149108:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078234.149110:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078234.149114:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4923 00000100:00080000:3.0:1769078234.149118:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078234.149122:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078234.149127:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078234.149131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078234/real 1769078234] req@ffff969e0a7fc380 x1855007923961728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078234.149141:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923962240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078234.149157:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923961728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078234.149171:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078234.149176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078234.149180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923961728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078234.149208:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961856:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078234.149229:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078234.149242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078234/real 1769078234] req@ffff969e0a7fd500 x1855007923961856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078234.149253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923961856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078234.149261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078234.149265:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078234.149268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078234.149274:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078234.149293:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078234.149302:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078234/real 1769078234] req@ffff969e0a7fc000 x1855007923961984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078234.149313:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923961984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078234.149322:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078234.149325:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078234.149328:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923961984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078234.149335:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078234.149353:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078234.149363:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078234/real 1769078234] req@ffff969e0a7fdc00 x1855007923962112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078234.149375:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923962112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078234.149383:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078234.149387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078234.149390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078234.149396:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078234.149413:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078234.149422:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078234/real 1769078234] req@ffff969e0a7ff800 x1855007923962240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078234.149434:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923962240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078234.149442:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078234.149446:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078234.149449:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078234.468703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078235.172912:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078235.492675:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078236.196881:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078236.516683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078237.220795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078237.540688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078238.244901:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078238.564730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078239.268720:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078239.268729:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078239.268733:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078239.268741:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4928 00000100:00080000:0.0:1769078239.268752:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078239.268757:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078239.268782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078239.268790:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923962368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078239.268822:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078239.268824:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078239.268826:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078239.268829:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4928 00000100:00080000:0.0:1769078239.268831:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078239.268836:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078239.268843:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078239.268852:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923962368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078239.268863:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923962496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078239.268874:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078239.268878:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078239.268880:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078239.268884:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4928 00000100:00080000:0.0:1769078239.268887:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078239.268892:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078239.268897:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078239.268909:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923962624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078239.268919:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078239.268921:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078239.268926:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078239.268928:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078239.268932:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4928 00000100:00080000:0.0:1769078239.268940:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078239.268944:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078239.268948:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078239.268956:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078239/real 1769078239] req@ffff969e06a63100 x1855007923962368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078239.268959:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923962752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078239.268986:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078239.268989:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078239.268991:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078239.268994:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4928 00000100:00080000:0.0:1769078239.268997:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078239.268998:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923962368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078239.269001:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078239.269006:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078239.269015:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923962880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078239.269017:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078239.269025:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078239.269030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923962368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078239.269056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078239.269079:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078239.269091:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078239/real 1769078239] req@ffff969e06a61880 x1855007923962496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078239.269103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923962496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078239.269116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078239.269119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078239.269122:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078239.269128:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078239.269145:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078239.269153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078239/real 1769078239] req@ffff969e06a61180 x1855007923962624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078239.269163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923962624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078239.269170:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078239.269173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078239.269176:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078239.269181:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078239.269197:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078239.269206:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078239/real 1769078239] req@ffff969e06a61f80 x1855007923962752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078239.269217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923962752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078239.269222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078239.269224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078239.269227:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078239.269231:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078239.269243:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078239.269249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078239/real 1769078239] req@ffff969e06a62680 x1855007923962880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078239.269256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923962880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078239.269262:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078239.269264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078239.269267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923962880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078239.588659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078240.292781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078240.612709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078241.316820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078241.636729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078242.340756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078242.660723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078243.364770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078243.684687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078244.388670:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078244.388678:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078244.388685:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078244.388694:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4933 00000100:00080000:0.0:1769078244.388705:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078244.388710:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078244.388714:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078244.388740:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923963008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078244.388763:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078244.388790:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1769078244.388790:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923963008:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078244.388792:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078244.388800:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4933 00000100:00080000:0.0:1769078244.388804:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078244.388807:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078244.388815:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078244.388826:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923963136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078244.388835:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078244.388837:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078244.388839:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078244.388842:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4933 00000100:00080000:0.0:1769078244.388844:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078244.388850:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078244.388855:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078244.388860:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078244.388868:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923963264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078244.388875:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078244.388879:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078244.388881:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078244.388886:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078244/real 1769078244] req@ffff969e06a60380 x1855007923963008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078244.388887:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4933 00000100:00080000:0.0:1769078244.388893:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078244.388902:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078244.388914:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923963008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078244.388922:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078244.388925:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078244.388932:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078244.388937:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923963008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078244.388939:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923963392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078244.388952:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078244.388955:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078244.388957:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078244.388962:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4933 00000100:00100000:1.0:1769078244.388964:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963136:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078244.388969:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078244.388976:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078244.388981:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078244.388987:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078244.388996:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923963520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078244.389003:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078244/real 1769078244] req@ffff969e06a61500 x1855007923963136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078244.389020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923963136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078244.389036:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078244.389040:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078244.389047:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078244.389054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078244.389072:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078244.389082:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078244/real 1769078244] req@ffff969e06a62d80 x1855007923963264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078244.389093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923963264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078244.389099:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078244.389102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078244.389104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078244.389109:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078244.389126:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078244.389135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078244/real 1769078244] req@ffff969e06a60700 x1855007923963392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078244.389144:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923963392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078244.389152:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078244.389154:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078244.389157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078244.389171:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078244.389186:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078244.389194:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078244/real 1769078244] req@ffff969e06a60a80 x1855007923963520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078244.389204:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923963520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078244.389219:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078244.389222:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078244.389225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078244.708698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078245.412760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078245.732677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078246.436772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078246.756659:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078247.460758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078247.780713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078248.484744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078248.804720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078249.508736:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078249.508754:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078249.508758:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078249.508767:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4938 00000100:00080000:3.0:1769078249.508778:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078249.508785:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078249.508792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078249.508828:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923963648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078249.508849:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078249.508851:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078249.508852:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078249.508855:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4938 00000100:00080000:3.0:1769078249.508857:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078249.508860:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078249.508868:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078249.508877:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923963776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078249.508887:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923963648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078249.508910:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078249.508913:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078249.508914:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078249.508918:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4938 00000100:00080000:3.0:1769078249.508920:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078249.508923:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078249.508927:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078249.508936:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923963904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078249.508941:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078249.508944:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078249.508945:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078249.508948:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4938 00000100:00080000:3.0:1769078249.508950:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078249.508952:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078249.508955:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078249.508961:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078249.508965:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923964032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078249.508971:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078249.508974:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078249.508976:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078249.508978:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4938 00000100:00080000:3.0:1769078249.508985:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078249.508988:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078249.508992:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078249.508994:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078249/real 1769078249] req@ffff969e0a7ffb80 x1855007923963648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078249.509003:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923964160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078249.509011:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923963648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078249.509020:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078249.509024:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078249.509027:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923963648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078249.509048:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078249.509061:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078249.509068:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078249/real 1769078249] req@ffff969e0a7fd880 x1855007923963776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078249.509075:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923963776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078249.509081:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078249.509083:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078249.509085:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078249.509088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078249.509096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078249.509101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078249/real 1769078249] req@ffff969e0a7fd180 x1855007923963904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078249.509108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923963904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078249.509113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078249.509114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078249.509116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923963904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078249.509119:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078249.509128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078249.509133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078249/real 1769078249] req@ffff969e0a7fe300 x1855007923964032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078249.509140:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923964032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078249.509145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078249.509146:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078249.509148:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078249.509150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078249.509158:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078249.509163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078249/real 1769078249] req@ffff969e0a7fca80 x1855007923964160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078249.509169:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923964160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078249.509174:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078249.509176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078249.509177:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078249.828727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078250.532799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078250.852720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078251.556790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078251.876712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078252.580835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078252.900720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078253.604795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078253.924689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078254.628724:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078254.628735:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078254.628741:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078254.628752:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4943 00000100:00080000:0.0:1769078254.628771:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078254.628778:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078254.628783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078254.628814:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923964288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078254.628840:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078254.628844:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078254.628846:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078254.628850:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4943 00000100:00080000:0.0:1769078254.628869:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078254.628874:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078254.628883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923964288:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078254.628884:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078254.628902:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923964416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078254.628913:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078254.628917:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078254.628919:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078254.628923:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4943 00000100:00080000:0.0:1769078254.628938:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078254.628943:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078254.628947:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078254.628954:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078254.628962:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923964544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078254.628972:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078254.628975:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078254.628978:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078254.628982:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4943 00000100:00080000:0.0:1769078254.628986:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078254.628992:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078254.628993:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078254/real 1769078254] req@ffff969e06a63800 x1855007923964288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078254.628998:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078254.629013:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923964672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078254.629021:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923964288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078254.629050:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078254.629054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078254.629055:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078254.629060:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078254.629061:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769078254.629067:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4943 00000100:00100000:1.0:1769078254.629070:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923964288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078254.629073:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078254.629080:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078254.629085:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078254.629096:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923964800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078254.629099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078254.629141:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078254.629155:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078254/real 1769078254] req@ffff969e06a60000 x1855007923964416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078254.629166:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923964416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078254.629175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078254.629178:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078254.629181:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078254.629186:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078254.629204:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078254.629214:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078254/real 1769078254] req@ffff969e06a60a80 x1855007923964544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078254.629223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923964544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078254.629231:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078254.629234:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078254.629237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078254.629242:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078254.629260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078254.629269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078254/real 1769078254] req@ffff969e06a60700 x1855007923964672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078254.629281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923964672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078254.629288:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078254.629291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078254.629294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078254.629307:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078254.629325:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078254.629335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078254/real 1769078254] req@ffff969e06a62d80 x1855007923964800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078254.629346:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923964800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078254.629353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078254.629356:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078254.629359:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923964800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078254.948720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078255.652740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078255.972716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078256.676866:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078256.996688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078257.700751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078258.020707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078258.725419:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078259.044654:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078259.749271:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078259.749631:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078259.749639:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078259.749644:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078259.749654:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4948 00000100:00080000:3.0:1769078259.749667:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078259.749674:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078259.749714:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923964928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078259.749744:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078259.749747:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078259.749749:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078259.749773:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4948 00000100:00080000:3.0:1769078259.749788:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078259.749797:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078259.749800:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923964928:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769078259.749811:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078259.749835:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923965056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078259.749848:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078259.749851:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078259.749853:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078259.749859:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4948 00000100:00080000:3.0:1769078259.749864:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078259.749868:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078259.749873:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078259.749887:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923965184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078259.749895:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078259.749909:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078259.749913:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078259.749916:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078259.749922:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4948 00000100:00080000:3.0:1769078259.749927:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078259.749932:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078259.749938:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078259.749950:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078259/real 1769078259] req@ffff969e0a7fdf80 x1855007923964928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078259.749957:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923965312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078259.749980:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923964928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078259.749982:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078259.749987:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078259.749989:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078259.749995:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4948 00000100:00080000:3.0:1769078259.750004:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078259.750013:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769078259.750014:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078259.750021:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078259.750024:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078259.750034:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923964928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078259.750043:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923965440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078259.750072:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078259.750095:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078259.750107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078259/real 1769078259] req@ffff969e0a7fea00 x1855007923965056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078259.750118:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923965056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078259.750127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078259.750130:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078259.750133:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078259.750139:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078259.750153:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078259.750161:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078259/real 1769078259] req@ffff969e0a7fc700 x1855007923965184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078259.750172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923965184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078259.750186:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078259.750189:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078259.750192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078259.750197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078259.750211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078259.750218:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078259/real 1769078259] req@ffff969e0a7fce00 x1855007923965312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078259.750228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923965312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078259.750417:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078259.750421:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078259.750424:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078259.750430:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078259.750450:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078259.750460:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078259/real 1769078259] req@ffff969e0a7fe680 x1855007923965440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078259.750482:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923965440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078259.750489:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078259.750492:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078259.750495:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078260.068697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078260.773694:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078261.092652:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078261.797171:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078262.116701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078262.820781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078263.140720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078263.844731:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078264.164707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078264.868689:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078264.868701:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078264.868706:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078264.868716:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4954 00000100:00080000:3.0:1769078264.868727:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078264.868734:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078264.868768:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923965568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078264.868775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078264.868803:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078264.868806:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078264.868822:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078264.868827:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4954 00000100:00080000:3.0:1769078264.868831:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078264.868835:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078264.868839:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923965568:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769078264.868845:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078264.868861:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923965696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078264.868872:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078264.868876:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078264.868877:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078264.868881:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4954 00000100:00080000:3.0:1769078264.868885:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078264.868888:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078264.868893:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078264.868903:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923965824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078264.868907:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078264.868916:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078264.868919:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078264.868921:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078264.868926:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4954 00000100:00080000:3.0:1769078264.868931:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078264.868934:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078264.868939:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078264.868943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078264/real 1769078264] req@ffff969e0a7fed80 x1855007923965568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078264.868959:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923965952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078264.868964:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923965568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078264.868979:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078264.868983:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078264.868984:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078264.868987:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078264.868992:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769078264.868993:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4954 00000100:00080000:3.0:1769078264.868999:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078264.869001:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923965568:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078264.869006:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078264.869012:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078264.869021:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923966080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078264.869031:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078264.869059:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078264.869071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078264/real 1769078264] req@ffff969e0a7fe680 x1855007923965696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078264.869083:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923965696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078264.869091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078264.869094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078264.869097:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078264.869103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078264.869121:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078264.869132:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078264/real 1769078264] req@ffff969e0a7fce00 x1855007923965824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078264.869142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923965824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078264.869150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078264.869152:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078264.869155:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078264.869160:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078264.869177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078264.869186:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078264/real 1769078264] req@ffff969e0a7fc700 x1855007923965952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078264.869195:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923965952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078264.869202:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078264.869205:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078264.869207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923965952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078264.869221:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078264.869239:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078264.869249:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078264/real 1769078264] req@ffff969e0a7fea00 x1855007923966080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078264.869258:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923966080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078264.869267:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078264.869300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078264.869303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078265.188721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078265.892776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078266.212728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078266.916798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078267.236714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078267.940775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078268.260711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078268.964774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078269.284719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078269.988712:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078269.988724:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078269.988728:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078269.988738:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4959 00000100:00080000:3.0:1769078269.988749:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078269.988756:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078269.988783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078269.988791:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923966208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078269.988816:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078269.988831:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078269.988833:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078269.988838:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4959 00000100:00080000:3.0:1769078269.988843:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078269.988848:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078269.988855:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923966208:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769078269.988858:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078269.988875:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923966336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078269.988885:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078269.988889:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078269.988891:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078269.988895:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4959 00000100:00080000:3.0:1769078269.988899:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078269.988903:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078269.988909:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078269.988920:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923966464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078269.988928:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078269.988933:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078269.988938:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078269.988951:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078269.988955:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4959 00000100:00080000:3.0:1769078269.988960:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078269.988964:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078269.988968:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078269/real 1769078269] req@ffff969e0a7fdf80 x1855007923966208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078269.988977:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078269.988992:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923966208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078269.988996:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923966592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078269.989012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078269.989015:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078269.989019:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:3.0:1769078269.989021:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078269.989025:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078269.989030:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4959 00000100:00080000:3.0:1769078269.989034:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078269.989039:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078269.989040:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923966208:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769078269.989053:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078269.989064:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923966720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078269.989068:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078269.989100:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078269.989117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078269/real 1769078269] req@ffff969e0a7fca80 x1855007923966336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078269.989130:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923966336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078269.989138:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078269.989141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078269.989145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078269.989151:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078269.989170:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078269.989180:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078269/real 1769078269] req@ffff969e0a7fe300 x1855007923966464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078269.989223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923966464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078269.989230:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078269.989233:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078269.989236:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078269.989242:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078269.989259:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078269.989269:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078269/real 1769078269] req@ffff969e0a7fd180 x1855007923966592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078269.989280:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923966592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078269.989288:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078269.989291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078269.989294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078269.989310:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078269.989328:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078269.989337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078269/real 1769078269] req@ffff969e0a7fd880 x1855007923966720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078269.989348:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923966720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078269.989356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078269.989359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078269.989362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078270.308693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078271.012761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078271.332696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078272.036762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078272.356683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078273.060794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078273.380748:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078274.084777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078274.404669:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078275.108764:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078275.108773:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078275.108776:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078275.108789:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4964 00000100:00080000:1.0:1769078275.108800:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769078275.108802:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078275.108806:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078275.108837:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923966848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078275.108863:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078275.108867:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078275.108868:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078275.108872:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4964 00000100:00080000:1.0:1769078275.108875:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078275.108879:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078275.108889:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078275.108901:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923966976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078275.108908:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923966848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:1.0:1769078275.108934:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078275.108938:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078275.108940:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078275.108945:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4964 00000100:00080000:1.0:1769078275.108949:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078275.108954:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078275.108959:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078275.108972:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923967104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078275.108979:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078275.108983:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1769078275.108985:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:1.0:1769078275.108985:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078275.108991:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4964 00000100:00080000:1.0:1769078275.108994:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078275.108997:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078275.109001:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078275.109009:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923967232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078275.109016:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078275.109018:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078275.109020:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078275.109022:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4964 00000100:00100000:0.0:1769078275.109027:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078275/real 1769078275] req@ffff969e10b62680 x1855007923966848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078275.109027:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078275.109035:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078275.109042:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078275.109056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923966848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078275.109061:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923967360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078275.109075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078275.109089:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078275.109094:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923966848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078275.109125:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078275.109142:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078275.109150:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078275/real 1769078275] req@ffff969e10b62a00 x1855007923966976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078275.109177:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923966976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078275.109183:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078275.109185:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078275.109187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923966976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078275.109191:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078275.109205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078275.109211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078275/real 1769078275] req@ffff969e10b60a80 x1855007923967104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078275.109217:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923967104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078275.109222:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078275.109224:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078275.109225:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078275.109229:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078275.109238:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078275.109243:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078275/real 1769078275] req@ffff969e10b61f80 x1855007923967232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078275.109250:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923967232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078275.109268:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078275.109270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078275.109272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078275.109275:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078275.109287:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078275.109293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078275/real 1769078275] req@ffff969e10b61500 x1855007923967360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078275.109299:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923967360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078275.109304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078275.109306:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078275.109307:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078275.428794:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078276.132791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078276.452682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078277.156794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078277.476742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078278.180817:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078278.500702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078279.204810:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078279.524784:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078280.228798:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078280.228814:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078280.228819:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078280.228833:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4969 00000100:00080000:3.0:1769078280.228870:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078280.228878:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078280.228925:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923967488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078280.228952:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078280.228965:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078280.228971:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078280.228973:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078280.228979:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4969 00000100:00080000:3.0:1769078280.228985:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078280.228990:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078280.229001:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078280.229024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923967488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078280.229030:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923967616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078280.229053:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078280.229057:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078280.229059:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078280.229064:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4969 00000100:00080000:3.0:1769078280.229068:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078280.229071:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078280.229077:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078280.229090:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923967744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078280.229100:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1769078280.229103:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078280.229114:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078280.229117:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078280.229122:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4969 00000100:00080000:3.0:1769078280.229127:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078280.229132:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078280.229139:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078280.229145:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078280/real 1769078280] req@ffff969e0a7ffb80 x1855007923967488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078280.229154:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923967872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078280.229167:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078280.229170:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078280.229172:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078280.229176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923967488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078280.229180:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4969 00000100:00080000:3.0:1769078280.229186:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078280.229192:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078280.229193:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:0.0:1769078280.229199:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000008:00100000:3.0:1769078280.229201:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078280.229205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923967488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078280.229217:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923968000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078280.229249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078280.229281:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078280.229291:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078280/real 1769078280] req@ffff969e0a7ff800 x1855007923967616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078280.229300:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923967616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078280.229306:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078280.229309:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078280.229312:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078280.229318:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078280.229337:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078280.229346:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078280/real 1769078280] req@ffff969e0a7fdc00 x1855007923967744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078280.229358:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923967744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078280.229366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078280.229369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078280.229371:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078280.229377:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078280.229397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078280.229407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078280/real 1769078280] req@ffff969e0a7fc000 x1855007923967872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078280.229425:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923967872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078280.229496:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078280.229499:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078280.229502:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923967872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078280.229506:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078280.229521:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078280.229529:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078280/real 1769078280] req@ffff969e0a7fd500 x1855007923968000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078280.229536:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923968000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078280.229541:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078280.229543:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078280.229545:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078280.548708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078281.252864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078281.572759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078282.276833:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078282.596731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078283.301013:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078283.620731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078284.324840:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078284.644766:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078285.348724:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078285.348734:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078285.348739:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078285.348750:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4974 00000100:00080000:0.0:1769078285.348761:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078285.348768:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078285.348801:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923968128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078285.348839:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078285.348841:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078285.348843:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078285.348846:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4974 00000100:00080000:0.0:1769078285.348848:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078285.348851:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078285.348858:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078285.348867:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923968256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078285.348874:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078285.348876:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078285.348877:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078285.348880:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4974 00000100:00080000:0.0:1769078285.348884:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078285.348887:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078285.348887:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:0.0:1769078285.348892:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078285.348909:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923968384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078285.348915:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078285.348917:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078285.348918:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078285.348920:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4974 00000100:00080000:0.0:1769078285.348942:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078285.348946:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078285.348951:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078285.348962:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923968512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078285.348968:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078285.348970:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078285.348972:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078285.348976:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4974 00000100:00080000:0.0:1769078285.348978:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078285.348984:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078285.348986:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923968128:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078285.348988:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078285.349000:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923968640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078285.349071:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078285.349110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078285/real 1769078285] req@ffff969e06a61500 x1855007923968128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078285.349128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923968128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078285.349140:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078285.349144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078285.349149:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923968128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078285.349156:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078285.349175:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078285.349185:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078285/real 1769078285] req@ffff969e06a60380 x1855007923968256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078285.349196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923968256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078285.349204:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078285.349207:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078285.349211:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078285.349217:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078285.349232:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078285.349240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078285/real 1769078285] req@ffff969e06a62680 x1855007923968384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078285.349251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923968384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078285.349259:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078285.349262:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078285.349265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078285.349293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078285.349308:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078285.349317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078285/real 1769078285] req@ffff969e06a61f80 x1855007923968512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078285.349328:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923968512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078285.349336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078285.349339:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078285.349343:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078285.349349:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078285.349362:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078285.349371:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078285/real 1769078285] req@ffff969e06a61180 x1855007923968640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078285.349381:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923968640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078285.349390:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078285.349393:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078285.349396:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078285.668708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078286.372809:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078286.692696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078287.396795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078287.716693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078288.420766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078288.740662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078289.444787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078289.764773:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078290.468704:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078290.468712:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078290.468716:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078290.468726:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4979 00000100:00080000:0.0:1769078290.468739:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078290.468746:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078290.468760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078290.468783:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923968768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078290.468804:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078290.468818:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078290.468819:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078290.468822:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4979 00000100:00080000:0.0:1769078290.468826:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078290.468829:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078290.468850:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078290.468861:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923968768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078290.468863:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923968896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078290.468874:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078290.468876:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078290.468877:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078290.468880:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4979 00000100:00080000:0.0:1769078290.468882:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078290.468885:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078290.468888:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078290.468897:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923969024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078290.468906:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078290.468909:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078290.468912:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078290.468915:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4979 00000100:00080000:0.0:1769078290.468919:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078290.468921:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078290.468926:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078290.468938:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078290.468949:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923969152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078290.468953:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078290/real 1769078290] req@ffff969e06a61880 x1855007923968768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078290.468961:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078290.468966:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078290.468968:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078290.468972:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4979 00000100:00100000:1.0:1769078290.468975:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923968768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078290.468977:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078290.468983:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769078290.468988:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769078290.468989:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078290.468995:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078290.468999:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923969280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078290.469000:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923968768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078290.469025:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078290.469042:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078290.469050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078290/real 1769078290] req@ffff969e06a63100 x1855007923968896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078290.469057:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923968896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078290.469063:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078290.469065:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078290.469067:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923968896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078290.469070:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078290.469080:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078290.469085:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078290/real 1769078290] req@ffff969e06a61180 x1855007923969024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078290.469091:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923969024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078290.469096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078290.469098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078290.469099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078290.469102:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078290.469111:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078290.469116:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078290/real 1769078290] req@ffff969e06a61f80 x1855007923969152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078290.469122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923969152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078290.469127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078290.469128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078290.469130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078290.469133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078290.469142:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078290.469146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078290/real 1769078290] req@ffff969e06a62680 x1855007923969280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078290.469152:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923969280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078290.469156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078290.469158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078290.469160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078290.788712:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078291.492777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078291.812709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078292.516811:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078292.836725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078293.540815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078293.860736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078294.564826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078294.884743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078295.588719:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078295.588733:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078295.588738:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078295.588782:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4984 00000100:00080000:0.0:1769078295.588793:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078295.588802:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078295.588826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078295.588846:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923969408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078295.588872:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078295.588877:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078295.588880:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078295.588884:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4984 00000100:00100000:1.0:1769078295.588901:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923969408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078295.588908:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078295.588915:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078295.588925:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078295.588947:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923969536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078295.588957:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078295.588960:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078295.588962:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078295.588967:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4984 00000100:00080000:0.0:1769078295.588971:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078295.588974:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078295.588979:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078295.588990:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923969664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078295.588999:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078295.589003:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078295.589006:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078295.589010:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4984 00000100:00080000:0.0:1769078295.589014:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078295.589019:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969536:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078295.589020:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078295.589029:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078295.589043:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923969792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078295.589058:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969664:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078295.589068:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078295.589073:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078295.589075:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078295.589081:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4984 00000100:00080000:0.0:1769078295.589086:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078295.589090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969792:192.168.204.143@tcp:8:kworker.0 00000100:00080000:0.0:1769078295.589094:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078295.589101:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078295.589114:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923969920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078295.589135:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078295.908684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078296.612744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078296.932757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078297.636815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078297.956668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078298.660936:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769078298.661056:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769078298.661100:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769078298.661125:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078298.661162:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078298.661170:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078298.661180:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078298.661190:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769078298.661215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078295/real 1769078298] req@ffff969e06a60a80 x1855007923969920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078298.661242:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923969920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078298.661255:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078298.661260:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078298.661265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078298.661272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078295/real 1769078298] req@ffff969e06a60700 x1855007923969792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078298.661285:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923969792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078298.661293:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078298.661296:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078298.661300:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078298.661309:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078295/real 1769078298] req@ffff969e06a61500 x1855007923969536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078298.661320:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923969536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078298.661330:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078298.661333:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078298.661337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078298.661343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078295/real 1769078298] req@ffff969e06a62d80 x1855007923969664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078298.661354:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923969664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078298.661361:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078298.661365:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078298.661368:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923969664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078298.661374:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078295/real 1769078298] req@ffff969e06a60380 x1855007923969408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078298.661385:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923969408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078298.661391:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078298.661394:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078298.661397:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923969408:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078298.980722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078299.684849:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078300.004739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078300.708699:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769078300.708706:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078300.708712:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078300.708717:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078300.708727:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4989 00000100:00080000:3.0:1769078300.708739:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078300.708746:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078300.708776:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923970048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078300.708803:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078300.708806:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078300.708808:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078300.708813:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4989 00000100:00080000:3.0:1769078300.708817:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078300.708821:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078300.708830:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078300.708845:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923970048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078300.708846:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923970176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078300.708862:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078300.708865:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078300.708867:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078300.708871:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4989 00000100:00080000:3.0:1769078300.708875:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078300.708879:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078300.708899:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078300.708912:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923970304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078300.708922:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078300.708926:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078300.708929:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078300.708931:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078300.708936:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4989 00000100:00080000:3.0:1769078300.708942:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078300.708947:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078300.708953:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078300.708964:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923970432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078300.708972:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078300/real 1769078300] req@ffff969e0a7fc380 x1855007923970048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078300.708976:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078300.708985:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078300.708988:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078300.708996:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4989 00000100:00080000:3.0:1769078300.709000:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078300.709006:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078300.709007:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923970048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078300.709012:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078300.709027:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923970560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078300.709045:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078300.709051:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078300.709055:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923970048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078300.709081:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078300.709104:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078300.709117:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078300/real 1769078300] req@ffff969e0a7fd500 x1855007923970176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078300.709129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923970176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078300.709137:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078300.709139:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078300.709142:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078300.709148:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078300.709165:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078300.709174:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078300/real 1769078300] req@ffff969e0a7fc000 x1855007923970304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078300.709183:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923970304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078300.709472:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078300.709477:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078300.709482:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078300.709489:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078300.709512:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078300.709524:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078300/real 1769078300] req@ffff969e0a7fdc00 x1855007923970432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078300.709535:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923970432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078300.709542:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078300.709545:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078300.709548:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078300.709554:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078300.709572:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078300.709581:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078300/real 1769078300] req@ffff969e0a7ff800 x1855007923970560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078300.709618:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923970560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078300.709626:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078300.709629:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078300.709632:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078301.028677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078301.732771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078302.052706:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078302.756759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078303.076689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078303.780765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078304.100705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078304.804760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078305.124705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078305.828722:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078305.828733:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078305.828737:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078305.828747:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 4995 00000100:00080000:0.0:1769078305.828759:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769078305.828761:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078305.828767:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078305.828802:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923970688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078305.828838:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078305.828842:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078305.828843:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078305.828848:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4995 00000100:00080000:0.0:1769078305.828853:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078305.828858:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078305.828865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923970688:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078305.828867:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078305.828884:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923970816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078305.828893:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078305.828896:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078305.828897:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078305.828901:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4995 00000100:00080000:0.0:1769078305.828904:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078305.828908:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078305.828912:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078305.828921:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923970944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078305.828929:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078305.828934:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078305.828935:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078305.828938:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078305.828943:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4995 00000100:00080000:0.0:1769078305.828952:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078305.828956:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078305.828962:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078305.828966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078305/real 1769078305] req@ffff969e06a60000 x1855007923970688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078305.828975:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923971072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078305.828985:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078305.828985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923970688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078305.828994:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078305.828999:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078305.829002:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078305.829006:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 4995 00000100:00080000:0.0:1769078305.829011:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078305.829012:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769078305.829016:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078305.829020:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923970688:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078305.829023:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078305.829037:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923971200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078305.829050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078305.829074:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078305.829083:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078305/real 1769078305] req@ffff969e06a63800 x1855007923970816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078305.829090:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923970816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078305.829096:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078305.829098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078305.829100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078305.829103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078305.829114:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078305.829119:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078305/real 1769078305] req@ffff969e06a60380 x1855007923970944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078305.829126:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923970944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078305.829134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078305.829136:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078305.829137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923970944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078305.829140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078305.829149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078305.829154:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078305/real 1769078305] req@ffff969e06a62d80 x1855007923971072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078305.829159:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923971072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078305.829164:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078305.829165:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078305.829167:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078305.829176:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078305.829185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078305.829190:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078305/real 1769078305] req@ffff969e06a61500 x1855007923971200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078305.829196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923971200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078305.829201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078305.829202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078305.829204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078306.148702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078306.852737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078307.172683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078307.876764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078308.196689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078308.900758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078309.220710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078309.924814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078310.244745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078310.948713:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078310.948725:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078310.948729:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078310.948740:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5000 00000100:00080000:3.0:1769078310.948752:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078310.948758:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078310.948777:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078310.948794:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923971328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078310.948820:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078310.948824:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078310.948826:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078310.948831:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5000 00000100:00080000:3.0:1769078310.948835:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078310.948840:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078310.948850:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078310.948858:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923971328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078310.948865:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923971456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078310.948877:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078310.948893:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078310.948896:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078310.948900:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5000 00000100:00080000:3.0:1769078310.948904:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078310.948908:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078310.948913:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078310.948927:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923971584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078310.948928:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078310.948940:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078310.948944:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078310.948946:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078310.948950:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5000 00000100:00080000:3.0:1769078310.948955:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078310.948959:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078310.948964:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078310.948966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078310/real 1769078310] req@ffff969e0a7ffb80 x1855007923971328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078310.948977:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923971712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078310.948987:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078310.948991:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923971328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078310.948991:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078310.948995:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078310.949002:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5000 00000100:00080000:3.0:1769078310.949009:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078310.949010:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078310.949015:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769078310.949018:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078310.949023:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923971328:192.168.204.143@tcp:250:kworker.0 00000008:00100000:3.0:1769078310.949027:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078310.949040:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923971840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078310.949050:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078310.949075:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078310.949089:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078310/real 1769078310] req@ffff969e0a7fd880 x1855007923971456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078310.949101:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923971456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078310.949108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078310.949111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078310.949114:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078310.949120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078310.949137:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078310.949147:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078310/real 1769078310] req@ffff969e0a7fd180 x1855007923971584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078310.949158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923971584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078310.949166:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078310.949169:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078310.949172:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078310.949177:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078310.949195:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078310.949205:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078310/real 1769078310] req@ffff969e0a7fe300 x1855007923971712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078310.949216:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923971712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078310.949223:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078310.949226:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078310.949229:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078310.949245:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078310.949263:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078310.949273:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078310/real 1769078310] req@ffff969e0a7fca80 x1855007923971840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078310.949284:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923971840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078310.949292:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078310.949295:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078310.949299:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923971840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078311.268671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078311.972760:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078312.292727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078312.996799:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078313.316673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078314.020766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078314.340740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078315.044805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078315.364684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078316.068717:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078316.068728:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078316.068733:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078316.068742:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5005 00000100:00080000:0.0:1769078316.068754:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078316.068761:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078316.068764:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078316.068799:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923971968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078316.068820:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078316.068839:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078316.068841:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078316.068844:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5005 00000100:00080000:0.0:1769078316.068848:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078316.068851:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078316.068861:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078316.068864:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923971968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078316.068872:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923972096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078316.068883:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078316.068885:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078316.068887:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078316.068892:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5005 00000100:00080000:0.0:1769078316.068895:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078316.068897:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078316.068900:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078316.068906:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923972224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078316.068912:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078316.068915:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078316.068916:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078316.068919:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5005 00000100:00080000:0.0:1769078316.068929:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078316.068931:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078316.068939:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078316.068949:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078316.068950:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923972352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078316.068959:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078316.068962:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078316.068963:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078316.068967:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5005 00000100:00080000:0.0:1769078316.068971:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078316.068974:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078316.068979:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078316.068987:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923972480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078316.069004:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078316/real 1769078316] req@ffff969e06a60700 x1855007923971968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078316.069020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923971968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078316.069028:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078316.069031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078316.069034:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923971968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078316.069053:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078316.069070:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078316.069078:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078316/real 1769078316] req@ffff969e06a60a80 x1855007923972096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078316.069085:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923972096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078316.069091:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078316.069093:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078316.069095:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078316.069100:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078316.069110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078316.069115:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078316/real 1769078316] req@ffff969e06a62680 x1855007923972224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078316.069122:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923972224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078316.069127:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078316.069128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078316.069130:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078316.069133:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078316.069151:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078316.069156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078316/real 1769078316] req@ffff969e06a61f80 x1855007923972352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078316.069163:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923972352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078316.069168:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078316.069170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078316.069171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078316.069175:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078316.069184:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078316.069189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078316/real 1769078316] req@ffff969e06a61180 x1855007923972480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078316.069196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923972480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078316.069201:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078316.069203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078316.069204:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078316.388730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078317.092793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078317.412703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078318.116740:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078318.436711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078319.140759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078319.460688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078320.164705:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078320.484759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078321.188711:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078321.188722:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078321.188726:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078321.188736:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5010 00000100:00080000:0.0:1769078321.188748:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078321.188755:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078321.188765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078321.188791:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923972608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078321.188812:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078321.188815:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078321.188817:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078321.188821:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5010 00000100:00080000:0.0:1769078321.188838:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078321.188842:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078321.188847:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923972608:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078321.188850:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078321.188862:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923972736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078321.188870:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078321.188873:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078321.188874:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078321.188877:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5010 00000100:00080000:0.0:1769078321.188880:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078321.188883:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078321.188887:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078321.188895:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923972864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078321.188901:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078321.188904:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078321.188906:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078321.188907:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078321.188909:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5010 00000100:00080000:0.0:1769078321.188912:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078321.188915:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078321.188919:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078321.188926:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923972992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078321.188942:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078321/real 1769078321] req@ffff969e06a63100 x1855007923972608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078321.188946:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078321.188949:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078321.188951:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078321.188954:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5010 00000100:00080000:0.0:1769078321.188958:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078321.188959:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923972608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078321.188961:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078321.188965:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078321.188969:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078321.188973:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078321.188974:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923973120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078321.188977:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923972608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078321.189000:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078321.189020:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078321.189030:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078321/real 1769078321] req@ffff969e06a61880 x1855007923972736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078321.189037:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923972736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078321.189047:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078321.189049:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078321.189051:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078321.189054:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972864:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078321.189065:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078321.189071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078321/real 1769078321] req@ffff969e06a61180 x1855007923972864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078321.189077:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923972864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078321.189082:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078321.189084:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078321.189086:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078321.189089:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078321.189098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078321.189102:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078321/real 1769078321] req@ffff969e06a61f80 x1855007923972992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078321.189108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923972992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078321.189113:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078321.189114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078321.189117:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923972992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078321.189120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078321.189128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078321.189133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078321/real 1769078321] req@ffff969e06a62680 x1855007923973120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078321.189138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923973120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078321.189144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078321.189145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078321.189146:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078321.508710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078322.212794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078322.532678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078323.237668:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078323.556721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078324.261397:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078324.580726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078325.285715:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078325.604737:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078326.309092:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078326.309102:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078326.309106:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078326.309119:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5015 00000100:00080000:3.0:1769078326.309134:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078326.309141:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078326.309182:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923973248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078326.309210:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078326.309246:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078326.309251:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078326.309273:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5015 00000100:00080000:3.0:1769078326.309279:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078326.309285:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078326.309296:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078326.309312:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923973376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078326.309322:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078326.309326:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078326.309328:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078326.309333:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5015 00000100:00080000:3.0:1769078326.309336:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078326.309341:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078326.309348:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078326.309354:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923973248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078326.309365:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923973504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078326.309378:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078326.309382:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078326.309384:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078326.309388:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5015 00000100:00080000:3.0:1769078326.309392:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078326.309396:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078326.309408:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078326.309423:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923973632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078326.309432:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078326.309436:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078326.309439:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078326.309443:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5015 00000100:00080000:3.0:1769078326.309448:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078326.309452:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769078326.309460:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:3.0:1769078326.309466:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078326.309483:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923973760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078326.309538:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078326/real 1769078326] req@ffff969e0a7fdf80 x1855007923973248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078326.309572:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923973248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078326.309584:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078326.309589:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078326.309628:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923973248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078326.309636:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078326.309662:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078326.309675:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078326/real 1769078326] req@ffff969e0a7fea00 x1855007923973376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078326.309686:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923973376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078326.309694:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078326.309698:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078326.309701:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078326.309732:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078326.309752:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078326.309763:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078326/real 1769078326] req@ffff969e0a7fc700 x1855007923973504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078326.309773:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923973504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078326.309780:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078326.309783:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078326.309786:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078326.309792:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078326.309810:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078326.309830:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078326/real 1769078326] req@ffff969e0a7fce00 x1855007923973632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078326.309841:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923973632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078326.309850:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078326.309853:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078326.309856:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078326.309862:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078326.309881:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078326.309891:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078326/real 1769078326] req@ffff969e0a7fe680 x1855007923973760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078326.309901:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923973760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078326.309909:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078326.309912:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078326.309915:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923973760:192.168.204.143@tcp:8:kworker.0 00010000:00010000:3.0:1769078326.310223:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078326.628713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078327.333724:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078327.652711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078328.356959:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078328.676703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078329.380786:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078329.700687:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078330.404778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078330.724711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078331.428721:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078331.428737:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078331.428741:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078331.428750:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5020 00010000:00010000:3.0:1769078331.428753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078331.428761:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078331.428768:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078331.428799:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923973888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078331.428821:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078331.428824:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078331.428826:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078331.428831:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5020 00000100:00080000:0.0:1769078331.428834:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078331.428839:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078331.428857:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923973888:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078331.428865:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078331.428883:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923974016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078331.428894:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078331.428897:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078331.428899:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078331.428903:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5020 00000100:00080000:0.0:1769078331.428908:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078331.428912:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078331.428917:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078331.428923:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078331.428937:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923974144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078331.428946:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078331.428948:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078331.428950:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078331.428953:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5020 00000100:00080000:0.0:1769078331.428956:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078331.428960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078331/real 1769078331] req@ffff969e06a60a80 x1855007923973888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078331.428961:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078331.428968:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078331.428985:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923974272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078331.428985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923973888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078331.429005:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078331.429009:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078331.429012:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078331.429018:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078331.429022:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769078331.429024:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5020 00000100:00100000:1.0:1769078331.429030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923973888:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078331.429033:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078331.429042:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078331.429047:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078331.429057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078331.429061:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923974400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078331.429083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078331.429097:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078331/real 1769078331] req@ffff969e06a60700 x1855007923974016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078331.429108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923974016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078331.429115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078331.429118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078331.429120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078331.429126:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078331.429140:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078331.429163:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078331/real 1769078331] req@ffff969e06a61500 x1855007923974144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078331.429175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923974144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078331.429182:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078331.429184:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078331.429187:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078331.429192:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078331.429208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078331.429216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078331/real 1769078331] req@ffff969e06a62d80 x1855007923974272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078331.429226:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923974272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078331.429233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078331.429235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078331.429237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078331.429250:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078331.429264:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078331.429272:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078331/real 1769078331] req@ffff969e06a60380 x1855007923974400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078331.429282:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923974400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078331.429289:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078331.429291:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078331.429294:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974400:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078331.748715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078332.452827:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078332.772685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078333.476784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078333.796729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078334.500803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078334.820697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078335.524816:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078335.844713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078336.548716:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078336.548727:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078336.548731:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078336.548741:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5025 00000100:00080000:0.0:1769078336.548753:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078336.548760:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078336.548772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078336.548797:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923974528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078336.548816:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078336.548820:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078336.548822:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078336.548826:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5025 00000100:00080000:0.0:1769078336.548829:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078336.548833:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078336.548842:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078336.548853:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923974656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078336.548854:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923974528:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078336.548882:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078336.548885:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078336.548887:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078336.548890:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5025 00000100:00080000:0.0:1769078336.548903:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078336.548906:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078336.548911:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078336.548918:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078336.548921:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923974784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078336.548928:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078336.548931:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078336.548932:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078336.548936:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5025 00000100:00080000:0.0:1769078336.548938:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078336.548941:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078336.548945:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078336.548951:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078336/real 1769078336] req@ffff969e06a63800 x1855007923974528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078336.548952:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923974912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078336.548961:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078336.548964:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078336.548965:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078336.548968:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5025 00000100:00100000:1.0:1769078336.548968:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923974528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078336.548976:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078336.548980:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769078336.548983:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1769078336.548984:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078336.548987:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078336.548990:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923974528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078336.548992:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923975040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078336.549012:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078336.549044:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078336.549053:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078336/real 1769078336] req@ffff969e06a60000 x1855007923974656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078336.549073:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923974656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078336.549079:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078336.549080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078336.549082:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974656:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078336.549086:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974784:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078336.549099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078336.549105:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078336/real 1769078336] req@ffff969e06a60380 x1855007923974784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078336.549111:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923974784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078336.549116:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078336.549118:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078336.549120:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078336.549123:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078336.549133:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078336.549137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078336/real 1769078336] req@ffff969e06a62d80 x1855007923974912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078336.549143:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923974912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078336.549148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078336.549150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078336.549151:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923974912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078336.549154:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078336.549162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078336.549167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078336/real 1769078336] req@ffff969e06a61500 x1855007923975040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078336.549173:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923975040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078336.549177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078336.549179:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078336.549180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078336.868647:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078337.572752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078337.892733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078338.596830:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078338.916697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078339.620738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078339.940730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078340.644793:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078340.964733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078341.668729:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078341.668742:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078341.668747:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078341.668759:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5030 00000100:00080000:0.0:1769078341.668771:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078341.668779:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078341.668792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078341.668825:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923975168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078341.668854:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078341.668858:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078341.668859:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078341.668864:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5030 00000100:00080000:0.0:1769078341.668868:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078341.668874:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078341.668884:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078341.668886:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923975168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078341.668901:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923975296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078341.668912:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078341.668935:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078341.668937:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078341.668942:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5030 00000100:00080000:0.0:1769078341.668947:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078341.668951:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078341.668953:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078341.668959:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078341.668991:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923975424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078341.669002:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078341.669007:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078341/real 1769078341] req@ffff969e06a60700 x1855007923975168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078341.669008:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078341.669012:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078341.669020:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5030 00000100:00080000:0.0:1769078341.669025:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078341.669030:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923975168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078341.669031:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078341.669040:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078341.669043:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078341.669047:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078341.669052:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923975168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078341.669063:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923975552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078341.669072:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078341.669076:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078341.669078:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078341.669081:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975296:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078341.669085:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5030 00000100:00080000:0.0:1769078341.669090:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078341.669094:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769078341.669103:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769078341.669105:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078341.669114:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078341/real 1769078341] req@ffff969e06a60a80 x1855007923975296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078341.669122:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923975680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078341.669127:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923975296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078341.669145:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078341.669149:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078341.669152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078341.669158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078341.669177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078341.669189:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078341/real 1769078341] req@ffff969e06a62680 x1855007923975424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078341.669202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923975424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078341.669218:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078341.669228:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078341.669230:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078341.669234:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078341.669250:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078341.669257:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078341/real 1769078341] req@ffff969e06a61f80 x1855007923975552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078341.669263:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923975552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078341.669269:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078341.669270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078341.669272:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078341.669281:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078341.669290:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078341.669304:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078341/real 1769078341] req@ffff969e06a61180 x1855007923975680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078341.669316:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923975680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078341.669324:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078341.669327:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078341.669331:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078341.988739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078342.692779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078343.012720:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078343.716794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078344.036711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078344.740783:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078345.060733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078345.764722:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078346.084723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078346.788687:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078346.788700:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078346.788705:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078346.788717:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5035 00000100:00080000:3.0:1769078346.788730:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078346.788739:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078346.788763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078346.788794:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923975808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078346.788823:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078346.788826:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078346.788828:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078346.788839:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5035 00000100:00100000:1.0:1769078346.788856:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923975808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078346.788864:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078346.788871:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078346.788882:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078346.788914:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923975936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078346.788927:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078346.788931:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078346.788933:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078346.788937:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5035 00000100:00080000:3.0:1769078346.788941:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078346.788944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078346.788949:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078346.788955:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078346.788967:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923976064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078346.788978:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078346.788982:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078346.788984:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078346.788988:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5035 00000100:00080000:3.0:1769078346.788992:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078346.788998:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078346/real 1769078346] req@ffff969e0a7fed80 x1855007923975808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078346.788998:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078346.789006:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078346.789023:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923976192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078346.789027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923975808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078346.789044:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078346.789053:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078346.789061:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923975808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078346.789061:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078346.789069:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078346.789072:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078346.789077:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5035 00000100:00080000:3.0:1769078346.789082:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078346.789086:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078346.789090:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975936:192.168.204.143@tcp:38:kworker.0 00000008:00100000:3.0:1769078346.789092:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078346.789109:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923976320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078346.789234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078346.789248:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078346/real 1769078346] req@ffff969e0a7fe680 x1855007923975936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078346.789256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923975936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078346.789261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078346.789263:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078346.789265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923975936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078346.789270:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078346.789283:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078346.789290:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078346/real 1769078346] req@ffff969e0a7fce00 x1855007923976064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078346.789296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923976064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078346.789301:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078346.789303:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078346.789305:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078346.789308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078346.789318:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078346.789323:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078346/real 1769078346] req@ffff969e0a7fc700 x1855007923976192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078346.789329:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923976192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078346.789334:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078346.789336:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078346.789338:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078346.789348:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078346.789356:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078346.789362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078346/real 1769078346] req@ffff969e0a7fea00 x1855007923976320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078346.789368:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923976320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078346.789374:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078346.789376:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078346.789378:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078347.108718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078347.812782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078348.132733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078348.836843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078349.156747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078349.860792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078350.180683:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078350.884992:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078351.204717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078351.908766:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769078351.908780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078351.908788:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078351.908794:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078351.908821:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5041 00000100:00080000:1.0:1769078351.908833:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078351.908841:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078351.908890:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61500 x1855007923976448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078351.908914:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078351.908916:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078351.908918:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078351.908921:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5041 00000100:00080000:1.0:1769078351.908924:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078351.908926:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078351.908935:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078351.908944:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b61f80 x1855007923976576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078351.908951:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078351.908953:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078351.908955:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078351.908958:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5041 00000100:00080000:1.0:1769078351.908961:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078351.908964:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078351.908969:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923976448:192.168.204.143@tcp:250:kworker.0 00000008:00100000:1.0:1769078351.908979:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078351.908994:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b60a80 x1855007923976704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078351.909006:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078351.909010:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1769078351.909012:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078351.909016:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5041 00000100:00080000:1.0:1769078351.909031:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078351.909035:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:1.0:1769078351.909041:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:0.0:1769078351.909049:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078351.909054:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62a00 x1855007923976832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078351.909065:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078351.909068:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1769078351.909070:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078351.909074:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5041 00000100:00080000:1.0:1769078351.909079:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078351.909083:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078351.909087:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078351/real 1769078351] req@ffff969e10b61500 x1855007923976448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1769078351.909090:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078351.909109:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e10b62680 x1855007923976960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078351.909112:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61500 x1855007923976448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078351.909136:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078351.909141:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078351.909161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923976448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078351.909167:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078351.909185:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078351.909197:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078351/real 1769078351] req@ffff969e10b61f80 x1855007923976576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078351.909205:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b61f80 x1855007923976576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078351.909210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078351.909213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078351.909216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078351.909240:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:0.0:1769078351.909272:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078351.909278:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078351/real 1769078351] req@ffff969e10b60a80 x1855007923976704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078351.909290:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b60a80 x1855007923976704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078351.909297:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078351.909300:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078351.909303:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078351.909309:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078351.909327:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078351.909337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078351/real 1769078351] req@ffff969e10b62a00 x1855007923976832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078351.909347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62a00 x1855007923976832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078351.909353:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078351.909355:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078351.909357:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078351.909361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:0.0:1769078351.909378:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078351.909385:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078351/real 1769078351] req@ffff969e10b62680 x1855007923976960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078351.909402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e10b62680 x1855007923976960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078351.909408:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078351.909410:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078351.909412:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e10b62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923976960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078352.228686:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078352.932828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078353.252757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078353.956815:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078354.276657:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078354.980828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078355.300719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078356.004881:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078356.324711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078357.028792:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078357.028806:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078357.028812:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078357.028826:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5046 00010000:00010000:2.0:1769078357.028834:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078357.028839:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078357.028849:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078357.028901:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923977088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078357.028940:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078357.028944:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078357.028946:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078357.028950:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5046 00000100:00080000:0.0:1769078357.028955:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078357.028960:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078357.028972:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078357.028992:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923977216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078357.028993:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923977088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078357.029044:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078357.029049:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078357.029051:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078357.029057:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5046 00000100:00080000:0.0:1769078357.029062:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078357.029066:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078357.029072:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078357.029089:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923977344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078357.029102:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078357.029109:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078357.029114:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078357.029117:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078357.029123:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5046 00000100:00080000:0.0:1769078357.029129:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078357.029134:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078357.029143:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078357.029156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078357/real 1769078357] req@ffff969e06a61880 x1855007923977088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078357.029161:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923977472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078357.029215:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078357.029218:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078357.029221:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078357.029227:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923977088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078357.029233:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5046 00000100:00080000:1.0:1769078357.029244:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078357.029246:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078357.029254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769078357.029256:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078357.029264:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923977088:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078357.029268:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078357.029285:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923977600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078357.029304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078357.029333:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078357.029345:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078357/real 1769078357] req@ffff969e06a63100 x1855007923977216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078357.029357:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923977216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078357.029366:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078357.029369:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078357.029372:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078357.029378:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078357.029398:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078357.029408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078357/real 1769078357] req@ffff969e06a61180 x1855007923977344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078357.029419:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923977344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078357.029427:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078357.029430:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078357.029433:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078357.029439:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078357.029456:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078357.029464:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078357/real 1769078357] req@ffff969e06a61f80 x1855007923977472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078357.029483:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923977472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078357.029492:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078357.029495:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078357.029498:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977472:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078357.029504:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078357.029523:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078357.029533:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078357/real 1769078357] req@ffff969e06a62680 x1855007923977600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078357.029551:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923977600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078357.029559:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078357.029563:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078357.029566:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078357.348708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078358.052797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078358.372719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078359.076750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078359.396723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078360.100794:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078360.421228:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078361.124779:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078361.444721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078362.148749:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078362.148759:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078362.148763:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078362.148772:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5051 00010000:00010000:3.0:1769078362.148778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078362.148783:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078362.148802:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078362.148844:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923977728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078362.148868:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078362.148871:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078362.148872:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078362.148875:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5051 00000100:00080000:0.0:1769078362.148878:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078362.148882:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078362.148891:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078362.148907:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923977728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078362.148908:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923977856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078362.148925:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078362.148951:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078362.148953:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078362.148956:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5051 00000100:00080000:0.0:1769078362.148961:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078362.148965:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078362.148969:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078362.148979:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923977984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078362.148986:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078362.148989:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078362.148990:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078362.148994:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5051 00000100:00080000:0.0:1769078362.148997:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078362.148999:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078362.149003:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078362.149008:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078362.149013:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923978112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078362.149024:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078362.149028:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078362.149029:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078362.149034:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5051 00000100:00080000:0.0:1769078362.149039:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078362.149041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977984:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078362.149047:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078362.149059:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078362.149062:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:0.0:1769078362.149077:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923978240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078362.149098:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078362.468750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078363.172806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078363.492702:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078364.196771:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078364.516725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078365.220778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078365.220835:0:2404:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769078365.220870:0:2404:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078365.220892:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078365.220924:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078365.220933:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078365.220942:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078365.220951:0:2404:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769078365.220962:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078362/real 1769078365] req@ffff969e06a62d80 x1855007923978112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078365.220994:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923978112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078365.221006:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078365.221011:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078365.221015:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078365.221023:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078362/real 1769078365] req@ffff969e06a60700 x1855007923977856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078365.221043:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923977856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078365.221051:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078365.221055:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078365.221058:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078365.221065:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078362/real 1769078365] req@ffff969e06a61500 x1855007923977984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078365.221076:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923977984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078365.221083:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078365.221086:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078365.221089:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923977984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078365.221096:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078362/real 1769078365] req@ffff969e06a60a80 x1855007923977728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078365.221108:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923977728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078365.221115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078365.221119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078365.221122:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923977728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078365.221146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078362/real 1769078365] req@ffff969e06a60380 x1855007923978240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078365.221158:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923978240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078365.221167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078365.221170:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078365.221174:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078365.540697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078366.244752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078366.564661:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078367.268763:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078367.268773:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078367.268777:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1769078367.268781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078367.268788:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5056 00000100:00080000:3.0:1769078367.268800:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078367.268807:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078367.268837:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923978368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078367.268867:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078367.268871:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078367.268873:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078367.268877:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5056 00000100:00080000:3.0:1769078367.268881:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078367.268885:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078367.268894:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078367.268905:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923978496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078367.268914:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078367.268917:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078367.268932:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078367.268937:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5056 00000100:00080000:3.0:1769078367.268940:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078367.268944:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078367.268949:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078367.268960:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923978624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078367.268970:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078367.268973:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078367.268975:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078367.268978:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5056 00000100:00080000:3.0:1769078367.268981:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078367.268984:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078367.268988:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078367.268998:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923978752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078367.269006:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078367.269009:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078367.269010:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078367.269013:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5056 00000100:00080000:3.0:1769078367.269017:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078367.269020:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078367.269024:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078367.269040:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923978880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078367.269121:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923978368:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078367.269194:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078367.269227:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078367/real 1769078367] req@ffff969e0a7fdf80 x1855007923978368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078367.269247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923978368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078367.269257:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078367.269261:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078367.269265:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923978368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078367.269272:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978496:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078367.269292:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078367.269303:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078367/real 1769078367] req@ffff969e0a7fca80 x1855007923978496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078367.269315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923978496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078367.269331:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078367.269334:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078367.269337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078367.269343:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978624:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078367.269359:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078367.269368:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078367/real 1769078367] req@ffff969e0a7fe300 x1855007923978624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078367.269379:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923978624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078367.269387:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078367.269390:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078367.269394:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078367.269400:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978752:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078367.269417:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078367.269427:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078367/real 1769078367] req@ffff969e0a7fd180 x1855007923978752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078367.269438:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923978752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078367.269445:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078367.269448:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078367.269453:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078367.269459:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978880:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078367.269475:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078367.269485:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078367/real 1769078367] req@ffff969e0a7fd880 x1855007923978880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078367.269495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923978880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078367.269504:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078367.269507:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078367.269509:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923978880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078367.588655:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078368.292735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078368.612668:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078369.316791:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078369.636738:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078370.340837:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078370.660721:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078371.364713:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078371.684677:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078372.388707:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078372.388719:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078372.388723:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078372.388734:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5061 00000100:00080000:3.0:1769078372.388745:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078372.388752:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078372.388788:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923979008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078372.388815:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078372.388819:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078372.388821:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078372.388826:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5061 00000100:00080000:3.0:1769078372.388830:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078372.388835:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078372.388844:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078372.388856:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923979136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078372.388866:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078372.388883:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078372.388885:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078372.388889:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5061 00000100:00080000:3.0:1769078372.388893:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078372.388897:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078372.388902:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078372.388914:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923979264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078372.388921:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078372.388924:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078372.388925:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078372.388927:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5061 00000100:00080000:3.0:1769078372.388930:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078372.388932:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078372.388935:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078372.388942:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923979392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078372.388947:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078372.388950:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078372.388951:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078372.388952:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5061 00000100:00080000:3.0:1769078372.388954:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078372.388955:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078372.388962:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078372.388968:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923979520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078372.389059:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923979008:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078372.389110:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078372.389137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078372/real 1769078372] req@ffff969e0a7ffb80 x1855007923979008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078372.389149:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923979008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078372.389155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078372.389158:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078372.389161:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923979008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078372.389165:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078372.389177:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078372.389184:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078372/real 1769078372] req@ffff969e0a7ff800 x1855007923979136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078372.389191:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923979136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078372.389195:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078372.389197:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078372.389199:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078372.389202:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078372.389211:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078372.389219:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078372/real 1769078372] req@ffff969e0a7fdc00 x1855007923979264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078372.389231:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923979264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078372.389238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078372.389241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078372.389245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078372.389251:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078372.389267:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078372.389274:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078372/real 1769078372] req@ffff969e0a7fc000 x1855007923979392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078372.389281:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923979392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078372.389287:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078372.389289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078372.389291:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078372.389295:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078372.389304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078372.389308:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078372/real 1769078372] req@ffff969e0a7fd500 x1855007923979520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078372.389315:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923979520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078372.389320:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078372.389321:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078372.389333:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979520:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769078372.389425:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078372.708704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078373.412766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078373.732658:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078374.436775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078374.756700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078375.460781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078375.780707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078376.484795:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078376.804709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078377.508710:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078377.508718:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078377.508721:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078377.508729:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5066 00000100:00080000:0.0:1769078377.508739:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078377.508745:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078377.508780:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923979648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078377.508784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078377.508806:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078377.508809:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078377.508810:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078377.508814:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5066 00000100:00080000:0.0:1769078377.508817:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078377.508820:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078377.508828:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078377.508839:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923979776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078377.508840:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923979648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078377.508854:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078377.508856:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078377.508857:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078377.508859:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5066 00000100:00080000:0.0:1769078377.508891:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078377.508896:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078377.508901:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078377.508906:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078377.508918:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923979904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078377.508930:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078377.508932:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078377.508934:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078377.508938:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5066 00000100:00100000:1.0:1769078377.508939:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078377/real 1769078377] req@ffff969e06a60000 x1855007923979648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078377.508941:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078377.508948:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078377.508961:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078377.508965:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923979648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078377.508976:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923980032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078377.508977:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078377.508987:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769078377.508993:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078377.508995:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923979648:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078377.509002:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078377.509005:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078377.509010:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5066 00000100:00080000:0.0:1769078377.509013:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078377.509016:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078377.509020:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078377.509022:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078377.509031:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923980160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078377.509046:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078377.509070:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078377/real 1769078377] req@ffff969e06a63800 x1855007923979776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078377.509084:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923979776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078377.509092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078377.509096:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078377.509099:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078377.509104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078377.509120:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078377.509129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078377/real 1769078377] req@ffff969e06a60380 x1855007923979904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078377.509139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923979904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078377.509147:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078377.509150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078377.509153:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923979904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078377.509158:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078377.509172:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078377.509181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078377/real 1769078377] req@ffff969e06a60a80 x1855007923980032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078377.509192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923980032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078377.509200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078377.509203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078377.509207:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078377.509220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078377.509234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078377.509242:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078377/real 1769078377] req@ffff969e06a61500 x1855007923980160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078377.509253:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923980160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078377.509261:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078377.509264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078377.509267:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980160:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078377.828717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078378.535445:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078378.852746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078379.556803:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078379.876798:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078380.580841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078380.900717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078381.604750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078381.924663:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078382.628707:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078382.628715:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078382.628719:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078382.628727:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5071 00000100:00080000:2.0:1769078382.628736:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078382.628742:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078382.628771:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923980288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1769078382.628776:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078382.628796:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078382.628799:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078382.628801:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078382.628803:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5071 00000100:00080000:2.0:1769078382.628806:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078382.628808:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078382.628814:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078382.628822:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923980416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078382.628829:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078382.628831:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1769078382.628841:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923980288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769078382.628852:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078382.628860:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5071 00000100:00080000:2.0:1769078382.628863:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078382.628865:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078382.628869:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078382.628878:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923980544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078382.628885:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078382.628887:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078382.628889:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078382.628891:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5071 00000100:00080000:2.0:1769078382.628896:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078382.628898:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078382.628901:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078382.628909:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078382.628910:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923980672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078382.628918:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078382.628920:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078382.628921:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078382.628923:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5071 00000100:00080000:2.0:1769078382.628926:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078382.628928:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078382.628930:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078382.628942:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923980800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078382.628949:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078382/real 1769078382] req@ffff969e0a46ea00 x1855007923980288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078382.628965:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923980288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078382.628980:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078382.628984:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078382.628988:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923980288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078382.628995:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078382.629018:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078382.629031:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078382/real 1769078382] req@ffff969e0a46d500 x1855007923980416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078382.629040:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923980416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078382.629046:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078382.629048:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078382.629050:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078382.629069:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078382.629082:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078382.629088:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078382/real 1769078382] req@ffff969e0a46ca80 x1855007923980544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078382.629094:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923980544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078382.629100:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078382.629102:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078382.629104:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078382.629108:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078382.629118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078382.629122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078382/real 1769078382] req@ffff969e0a46df80 x1855007923980672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078382.629129:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923980672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078382.629134:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078382.629135:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078382.629137:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078382.629140:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078382.629149:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078382.629153:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078382/real 1769078382] req@ffff969e0a46f100 x1855007923980800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078382.629160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923980800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078382.629165:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078382.629166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078382.629168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923980800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078382.948722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078383.652831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078383.972667:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078384.676782:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078384.996743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078385.700789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078386.020722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078386.724750:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078387.044695:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078387.748686:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078387.748696:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078387.748700:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078387.748710:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5076 00000100:00080000:2.0:1769078387.748721:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078387.748727:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769078387.748749:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078387.748763:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923980928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078387.748784:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078387.748787:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078387.748788:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078387.748792:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5076 00000100:00080000:2.0:1769078387.748795:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078387.748798:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078387.748806:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078387.748816:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923980928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078387.748819:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923981056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078387.748830:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078387.748832:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078387.748833:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078387.748836:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5076 00000100:00080000:2.0:1769078387.748839:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078387.748855:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078387.748859:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078387.748868:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923981184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078387.748876:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078387.748880:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1769078387.748883:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078387.748884:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078387.748891:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5076 00000100:00080000:2.0:1769078387.748895:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078387.748898:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078387.748902:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078387.748910:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923981312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078387.748917:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078387.748920:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078387.748921:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078387.748924:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078387/real 1769078387] req@ffff969e0a46ed80 x1855007923980928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078387.748927:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5076 00000100:00080000:2.0:1769078387.748932:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078387.748940:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078387.748947:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078387.748951:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923980928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078387.748961:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1769078387.748963:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923981440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078387.748969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078387.748979:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923980928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078387.749012:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981056:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078387.749037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078387.749050:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078387/real 1769078387] req@ffff969e0a46e300 x1855007923981056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078387.749060:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923981056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078387.749066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078387.749069:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078387.749071:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078387.749075:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078387.749088:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078387.749095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078387/real 1769078387] req@ffff969e0a46f100 x1855007923981184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078387.749102:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923981184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078387.749112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078387.749114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078387.749116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078387.749120:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078387.749130:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078387.749135:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078387/real 1769078387] req@ffff969e0a46df80 x1855007923981312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078387.749142:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923981312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078387.749148:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078387.749150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078387.749152:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078387.749155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078387.749164:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078387.749169:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078387/real 1769078387] req@ffff969e0a46ca80 x1855007923981440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078387.749175:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923981440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078387.749181:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078387.749183:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078387.749186:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078388.068700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078388.772735:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078389.092646:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078389.797334:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078390.116707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078390.821330:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078391.140763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078391.844874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078392.164743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078392.869531:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078392.870070:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078392.870080:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078392.870085:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078392.870097:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5082 00000100:00080000:0.0:1769078392.870109:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078392.870127:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078392.870186:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923981568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078392.870224:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078392.870227:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078392.870229:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078392.870234:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5082 00000100:00080000:0.0:1769078392.870238:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078392.870242:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078392.870252:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078392.870265:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923981696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078392.870274:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078392.870277:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078392.870279:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078392.870283:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5082 00000100:00080000:0.0:1769078392.870287:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078392.870291:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078392.870313:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078392.870323:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923981824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078392.870332:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078392.870335:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078392.870337:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078392.870340:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5082 00000100:00080000:0.0:1769078392.870343:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078392.870347:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078392.870351:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078392.870358:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923981952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078392.870366:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078392.870369:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078392.870370:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078392.870373:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5082 00000100:00080000:0.0:1769078392.870375:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078392.870379:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078392.870382:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078392.870390:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923982080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078392.870692:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923981568:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769078392.870759:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078392.870794:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078392/real 1769078392] req@ffff969e06a60700 x1855007923981568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078392.870812:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923981568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078392.870822:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078392.870827:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078392.870830:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923981568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078392.870838:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078392.870865:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078392.870880:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078392/real 1769078392] req@ffff969e06a62d80 x1855007923981696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078392.870892:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923981696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078392.870902:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078392.870907:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078392.870912:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078392.870919:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078392.870948:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078392.870958:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078392/real 1769078392] req@ffff969e06a62680 x1855007923981824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078392.870969:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923981824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078392.870976:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078392.870979:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078392.870981:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078392.870987:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078392.871003:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078392.871011:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078392/real 1769078392] req@ffff969e06a61f80 x1855007923981952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078392.871020:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923981952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078392.871028:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078392.871031:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078392.871034:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923981952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078392.871039:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078392.871055:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078392.871064:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078392/real 1769078392] req@ffff969e06a61180 x1855007923982080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078392.871086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923982080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078392.871094:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078392.871097:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078392.871100:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078393.188665:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078393.893921:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078394.212716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078394.918653:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078395.236713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078395.941406:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078396.261773:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078396.964864:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078397.284872:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078397.989038:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078397.989081:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078397.989101:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078397.989109:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5087 00000100:00080000:0.0:1769078397.989120:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078397.989127:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078397.989158:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923982208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078397.989186:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078397.989189:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078397.989191:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078397.989195:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5087 00000100:00080000:0.0:1769078397.989199:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078397.989202:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078397.989211:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078397.989221:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923982336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078397.989229:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078397.989232:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078397.989233:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078397.989237:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5087 00000100:00080000:0.0:1769078397.989239:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078397.989242:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078397.989246:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078397.989256:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923982464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078397.989275:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078397.989278:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078397.989280:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078397.989283:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5087 00000100:00080000:0.0:1769078397.989287:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078397.989290:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078397.989295:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078397.989304:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923982592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078397.989312:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078397.989315:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078397.989316:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078397.989319:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5087 00000100:00080000:0.0:1769078397.989322:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078397.989325:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078397.989329:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078397.989336:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923982720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078397.989431:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923982208:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078397.989495:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078397.989529:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078397/real 1769078397] req@ffff969e06a63100 x1855007923982208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078397.989546:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923982208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078397.989556:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078397.989560:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078397.989564:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923982208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078397.989571:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078397.989609:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078397.989634:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078397/real 1769078397] req@ffff969e06a61880 x1855007923982336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078397.989647:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923982336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078397.989656:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078397.989659:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078397.989662:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078397.989668:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078397.989687:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078397.989697:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078397/real 1769078397] req@ffff969e06a61180 x1855007923982464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078397.989708:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923982464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078397.989727:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078397.989731:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078397.989734:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078397.989740:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078397.989759:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078397.989770:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078397/real 1769078397] req@ffff969e06a61f80 x1855007923982592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078397.989780:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923982592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078397.989788:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078397.989791:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078397.989794:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078397.989800:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078397.989815:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078397.989823:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078397/real 1769078397] req@ffff969e06a62680 x1855007923982720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078397.989834:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923982720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078397.989842:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078397.989845:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078397.989848:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982720:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769078397.990004:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078398.309143:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078399.012797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078399.332699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078400.036770:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078400.356726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078401.062051:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078401.380725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078402.084801:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078402.404714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078403.108805:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078403.108817:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078403.108821:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078403.108831:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5092 00000100:00080000:2.0:1769078403.108842:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078403.108850:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078403.108885:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923982848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078403.108910:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078403.108914:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078403.108931:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078403.108936:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5092 00000100:00080000:2.0:1769078403.108954:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078403.108958:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078403.108968:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078403.108981:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923982976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078403.108990:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078403.108993:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078403.108995:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078403.108998:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5092 00000100:00080000:2.0:1769078403.109001:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078403.109005:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078403.109010:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078403.109020:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923983104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078403.109027:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078403.109031:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078403.109032:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078403.109036:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5092 00000100:00080000:2.0:1769078403.109038:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078403.109041:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078403.109045:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078403.109062:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923983232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078403.109071:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078403.109074:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078403.109076:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078403.109079:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5092 00000100:00080000:2.0:1769078403.109083:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078403.109087:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078403.109091:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078403.109101:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923983360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078403.110115:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923982848:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078403.110181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078403.110216:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078403/real 1769078403] req@ffff969e0a46d500 x1855007923982848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078403.110234:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923982848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078403.110243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078403.110247:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078403.110251:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923982848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078403.110258:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078403.110280:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078403.110292:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078403/real 1769078403] req@ffff969e0a46ea00 x1855007923982976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078403.110304:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923982976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078403.110311:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078403.110315:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078403.110318:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923982976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078403.110324:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078403.110342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078403.110352:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078403/real 1769078403] req@ffff969e0a46c380 x1855007923983104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078403.110363:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923983104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078403.110369:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078403.110372:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078403.110375:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078403.110380:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078403.110397:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078403.110407:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078403/real 1769078403] req@ffff969e0a46ce00 x1855007923983232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078403.110985:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923983232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078403.110991:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078403.110994:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078403.110996:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078403.111001:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078403.111017:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078403.111025:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078403/real 1769078403] req@ffff969e0a46f800 x1855007923983360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078403.111032:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923983360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078403.111037:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078403.111039:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078403.111041:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983360:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769078403.111984:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078403.428716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078404.132862:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078404.453670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078405.157310:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078405.476716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078406.180967:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078406.500714:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078407.205523:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078407.524763:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078408.231584:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078408.231967:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078408.231976:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078408.231981:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078408.231993:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5097 00000100:00080000:0.0:1769078408.232029:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078408.232038:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078408.232082:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923983488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078408.232116:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078408.232120:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078408.232122:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078408.232127:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5097 00000100:00080000:0.0:1769078408.232130:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078408.232134:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078408.232146:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078408.232157:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923983616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078408.232166:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078408.232169:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078408.232171:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078408.232175:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5097 00000100:00080000:0.0:1769078408.232178:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078408.232182:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078408.232186:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078408.232196:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923983744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078408.232205:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078408.232214:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078408.232216:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078408.232220:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5097 00000100:00080000:0.0:1769078408.232225:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078408.232228:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078408.232234:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078408.232244:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923983872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078408.232253:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078408.232257:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078408.232258:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078408.232262:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5097 00000100:00080000:0.0:1769078408.232265:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078408.232269:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078408.232273:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078408.232284:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923984000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078408.232371:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923983488:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078408.232445:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078408.232485:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078408/real 1769078408] req@ffff969e06a62d80 x1855007923983488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078408.232503:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923983488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078408.232513:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078408.232518:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078408.232522:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923983488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078408.232529:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078408.232552:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078408.232563:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078408/real 1769078408] req@ffff969e06a60700 x1855007923983616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078408.232575:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923983616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078408.232610:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078408.232614:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078408.232617:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078408.232623:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078408.232643:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078408.232654:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078408/real 1769078408] req@ffff969e06a61500 x1855007923983744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078408.232666:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923983744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078408.232674:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078408.232677:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078408.232680:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078408.232686:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078408.232706:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078408.232716:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078408/real 1769078408] req@ffff969e06a60a80 x1855007923983872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078408.232727:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923983872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078408.232737:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078408.232740:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078408.232743:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923983872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078408.232749:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078408.232767:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078408.232778:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078408/real 1769078408] req@ffff969e06a60380 x1855007923984000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078408.232789:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923984000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078408.232797:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078408.232800:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078408.232803:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078408.552012:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078409.252787:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078409.572764:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078410.276773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078410.596719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078411.300831:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078411.620759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078412.324875:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078412.644743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078413.348702:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078413.348710:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078413.348715:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078413.348724:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5102 00000100:00080000:0.0:1769078413.348733:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078413.348739:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078413.348784:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923984128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078413.348806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078413.348831:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078413.348836:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078413.348838:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078413.348841:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5102 00000100:00080000:0.0:1769078413.348844:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078413.348847:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078413.348854:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078413.348864:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923984256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078413.348871:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078413.348873:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078413.348874:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078413.348876:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5102 00000100:00080000:0.0:1769078413.348878:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078413.348880:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078413.348883:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078413.348889:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923984384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078413.348894:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078413.348896:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078413.348897:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078413.348901:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5102 00000100:00080000:0.0:1769078413.348912:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078413.348915:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078413.348920:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078413.348931:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923984512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078413.348940:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078413.348944:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078413.348945:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078413.348957:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5102 00000100:00080000:0.0:1769078413.348960:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078413.348962:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078413.348965:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078413.348973:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923984640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078413.349029:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923984128:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078413.349094:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078413.349130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078413/real 1769078413] req@ffff969e06a63800 x1855007923984128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078413.349147:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923984128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078413.349157:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078413.349162:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078413.349165:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923984128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078413.349172:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078413.349190:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078413.349198:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078413/real 1769078413] req@ffff969e06a60000 x1855007923984256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078413.349206:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923984256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078413.349211:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078413.349213:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078413.349215:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078413.349218:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078413.349228:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078413.349233:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078413/real 1769078413] req@ffff969e06a60380 x1855007923984384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078413.349239:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923984384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078413.349243:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078413.349245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078413.349247:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078413.349249:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078413.349258:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078413.349263:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078413/real 1769078413] req@ffff969e06a60a80 x1855007923984512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078413.349270:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923984512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078413.349279:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078413.349282:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078413.349285:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078413.349301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078413.349319:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078413.349329:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078413/real 1769078413] req@ffff969e06a61500 x1855007923984640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078413.349336:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923984640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078413.349342:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078413.349344:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078413.349346:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078413.668693:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078414.372798:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078414.692700:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078415.396975:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078415.716709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078416.420772:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078416.740699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078417.444828:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078417.764758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078418.468731:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078418.468755:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078418.468758:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078418.468767:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5107 00000100:00080000:2.0:1769078418.468778:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078418.468784:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:3.0:1769078418.468790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078418.468821:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007923984768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078418.468848:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078418.468852:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078418.468854:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078418.468858:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5107 00000100:00080000:2.0:1769078418.468862:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078418.468867:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923984768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769078418.468867:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078418.468877:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078418.468890:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007923984896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078418.468901:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078418.468904:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078418.468906:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078418.468909:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5107 00000100:00080000:2.0:1769078418.468913:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078418.468917:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078418.468922:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078418.468935:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923985024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078418.468937:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078418.468952:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078418.468955:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078418.468956:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078418.468960:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5107 00000100:00080000:2.0:1769078418.468964:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078418.468968:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078418.468971:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078418/real 1769078418] req@ffff969e0a46f480 x1855007923984768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769078418.468975:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078418.468989:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007923984768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078418.468993:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923985152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078418.469001:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078418.469008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:2.0:1769078418.469009:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078418.469014:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923984768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:2.0:1769078418.469014:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078418.469017:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078418.469022:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5107 00000100:00080000:2.0:1769078418.469025:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078418.469028:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078418.469031:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078418.469037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078418.469041:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923985280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078418.469056:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078418.469066:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078418/real 1769078418] req@ffff969e0a46d880 x1855007923984896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078418.469074:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007923984896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078418.469080:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078418.469082:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078418.469084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923984896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078418.469088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078418.469096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078418.469101:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078418/real 1769078418] req@ffff969e0a46f800 x1855007923985024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078418.469107:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923985024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078418.469112:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078418.469114:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078418.469116:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078418.469118:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078418.469127:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078418.469131:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078418/real 1769078418] req@ffff969e0a46ce00 x1855007923985152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078418.469138:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923985152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078418.469142:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078418.469144:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078418.469145:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078418.469155:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078418.469162:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078418.469167:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078418/real 1769078418] req@ffff969e0a46c380 x1855007923985280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078418.469172:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923985280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078418.469177:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078418.469179:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078418.469180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078418.788645:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078419.492835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078419.812651:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078420.516789:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078420.836727:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078421.540995:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078421.860719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078422.565172:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078422.884747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078423.588870:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078423.588912:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078423.588920:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078423.588924:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078423.588934:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5112 00000100:00080000:0.0:1769078423.588944:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078423.588950:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078423.588988:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923985408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078423.589011:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078423.589022:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078423.589024:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078423.589030:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5112 00000100:00080000:0.0:1769078423.589034:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078423.589039:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078423.589043:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923985408:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078423.589049:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078423.589064:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923985536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078423.589076:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078423.589079:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078423.589081:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078423.589085:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5112 00000100:00080000:0.0:1769078423.589089:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078423.589092:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078423.589097:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078423.589098:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078423.589110:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923985664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078423.589127:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078423.589130:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078423/real 1769078423] req@ffff969e06a60700 x1855007923985408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078423.589132:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078423.589136:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078423.589143:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5112 00000100:00080000:0.0:1769078423.589148:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078423.589152:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078423.589160:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923985408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078423.589163:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078423.589177:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923985792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078423.589200:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078423.589204:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078423.589206:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078423.589206:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078423.589217:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5112 00000100:00080000:1.0:1769078423.589217:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00080000:0.0:1769078423.589222:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078423.589226:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923985408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078423.589229:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078423.589236:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078423.589246:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923985920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078423.589254:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985536:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078423.589274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078423.589284:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078423/real 1769078423] req@ffff969e06a62d80 x1855007923985536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078423.589296:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923985536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078423.589304:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078423.589307:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078423.589310:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078423.589315:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078423.589334:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078423.589343:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078423/real 1769078423] req@ffff969e06a62680 x1855007923985664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078423.589352:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923985664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078423.589357:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078423.589359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078423.589361:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078423.589367:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078423.589385:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078423.589395:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078423/real 1769078423] req@ffff969e06a61f80 x1855007923985792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078423.589406:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923985792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078423.589414:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078423.589417:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078423.589419:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078423.589434:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078423.589452:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078423.589462:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078423/real 1769078423] req@ffff969e06a61180 x1855007923985920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078423.589473:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923985920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078423.589481:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078423.589485:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078423.589488:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923985920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078423.908703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078424.612952:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078424.933111:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078425.636837:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078425.959910:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078426.660732:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078426.980723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078427.684706:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078428.004662:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078428.708748:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078428.708762:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078428.708767:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078428.708777:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5117 00000100:00080000:0.0:1769078428.708791:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078428.708798:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078428.708840:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923986048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078428.708841:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078428.708878:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078428.708882:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078428.708884:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078428.708888:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5117 00000100:00080000:0.0:1769078428.708907:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078428.708912:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078428.708920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923986048:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078428.708923:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078428.708941:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923986176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078428.708953:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078428.708957:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078428.708959:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078428.708964:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5117 00000100:00080000:0.0:1769078428.708968:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078428.708973:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078428.708978:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078428.708992:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923986304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078428.709004:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078428.709007:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078428.709009:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078428.709013:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986176:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078428.709024:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5117 00000100:00080000:0.0:1769078428.709030:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078428.709035:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078428.709040:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078428.709054:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923986432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078428.709057:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986304:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078428.709091:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078428.709093:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986432:192.168.204.143@tcp:8:kworker.0 00000100:00080000:0.0:1769078428.709097:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078428.709101:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078428.709107:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5117 00000100:00080000:0.0:1769078428.709113:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078428.709118:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078428.709125:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078428.709140:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923986560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078428.709194:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078429.028676:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078429.732780:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078430.052713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078430.758697:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078431.076703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078431.780724:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1769078431.780793:0:2406:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:3.0:1769078431.780835:0:2406:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1769078431.780859:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769078431.780896:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769078431.780905:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769078431.780913:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:3.0:1769078431.780925:0:2406:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769078431.780943:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078428/real 1769078431] req@ffff969e06a61f80 x1855007923986432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078431.780971:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923986432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078431.780984:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078431.780989:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078431.780994:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078431.781001:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078428/real 1769078431] req@ffff969e06a63100 x1855007923986176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078431.781014:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923986176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078431.781021:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078431.781026:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078431.781030:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078431.781037:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078428/real 1769078431] req@ffff969e06a61180 x1855007923986304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078431.781048:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923986304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078431.781054:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078431.781057:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078431.781060:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078431.781079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078428/real 1769078431] req@ffff969e06a61880 x1855007923986048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078431.781093:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923986048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078431.781101:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078431.781105:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078431.781108:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923986048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078431.781136:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078428/real 1769078431] req@ffff969e06a62680 x1855007923986560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078431.781148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923986560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078431.781155:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078431.781165:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078431.781168:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078432.100707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078432.804759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078433.124731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078433.828814:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078433.829117:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078433.829124:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078433.829128:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078433.829137:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5123 00000100:00080000:0.0:1769078433.829148:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078433.829156:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078433.829190:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923986688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078433.829222:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078433.829226:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078433.829229:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078433.829234:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5123 00000100:00080000:0.0:1769078433.829239:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078433.829244:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078433.829270:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078433.829285:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923986816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078433.829298:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078433.829302:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078433.829304:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078433.829310:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5123 00000100:00080000:0.0:1769078433.829317:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078433.829323:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078433.829328:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078433.829341:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923986944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078433.829353:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923986688:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078433.829358:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078433.829369:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078433.829371:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078433.829377:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5123 00000100:00080000:0.0:1769078433.829382:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078433.829387:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078433.829395:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078433.829411:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923987072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078433.829433:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078433.829437:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078433.829447:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078433.829451:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5123 00000100:00080000:0.0:1769078433.829482:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078433.829488:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078433.829497:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078433.829507:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:0.0:1769078433.829522:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923987200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078433.829564:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078433/real 1769078433] req@ffff969e06a62d80 x1855007923986688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078433.831449:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923986688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078433.831472:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078433.831480:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078433.831486:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923986688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078433.831501:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986816:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078433.831570:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078433.831619:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078433/real 1769078433] req@ffff969e06a60700 x1855007923986816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078433.831636:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923986816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078433.831644:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078433.831647:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078433.831650:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078433.831680:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986944:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078433.831699:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078433.831706:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078433/real 1769078433] req@ffff969e06a61500 x1855007923986944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078433.831715:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923986944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078433.831720:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078433.831723:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078433.831726:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923986944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078433.831733:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987072:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078433.831752:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078433.831762:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078433/real 1769078433] req@ffff969e06a60a80 x1855007923987072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078433.831773:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923987072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078433.831781:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078433.831785:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078433.831788:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078433.831795:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987200:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078433.831820:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078433.831827:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078433/real 1769078433] req@ffff969e06a60380 x1855007923987200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078433.831839:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923987200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078433.831846:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078433.831848:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078433.831850:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078434.148745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078434.853773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078435.172760:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078435.876829:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078436.196756:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078436.900790:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078437.221490:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078437.924985:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078438.247129:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078438.951413:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078438.951423:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078438.951426:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078438.951434:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5128 00000100:00080000:3.0:1769078438.951443:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078438.951448:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078438.951479:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923987328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078438.951499:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078438.951501:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078438.951503:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078438.951505:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5128 00000100:00080000:3.0:1769078438.951507:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078438.951510:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078438.951517:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078438.951526:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923987456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078438.951532:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078438.951547:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078438.951548:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078438.951551:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5128 00000100:00080000:3.0:1769078438.951554:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078438.951556:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078438.951560:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078438.951568:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923987328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078438.951569:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923987584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078438.951583:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078438.951585:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078438.951586:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078438.951603:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5128 00000100:00080000:3.0:1769078438.951606:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078438.951608:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078438.951611:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078438.951618:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923987712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078438.951624:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078438.951627:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078438.951628:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078438.951630:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5128 00000100:00080000:3.0:1769078438.951633:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078438.951635:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078438.951642:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078438.951649:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923987840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078438.951659:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078438.951695:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078438/real 1769078438] req@ffff969e0a7fc380 x1855007923987328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078438.951714:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923987328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078438.951724:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078438.951728:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078438.951732:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923987328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078438.951739:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078438.951757:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078438.951768:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078438/real 1769078438] req@ffff969e0a7fd500 x1855007923987456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078438.951779:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923987456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078438.951787:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078438.951790:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078438.951794:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078438.951815:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078438.951830:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078438.951839:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078438/real 1769078438] req@ffff969e0a7fc000 x1855007923987584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078438.951850:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923987584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078438.951858:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078438.951860:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078438.951863:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078438.951869:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078438.951883:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078438.951891:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078438/real 1769078438] req@ffff969e0a7fdc00 x1855007923987712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078438.951901:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923987712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078438.951915:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078438.951918:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078438.951921:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078438.951927:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078438.951940:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078438.951948:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078438/real 1769078438] req@ffff969e0a7ff800 x1855007923987840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078438.951959:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923987840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078438.951967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078438.951971:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078438.951974:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923987840:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769078438.957422:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078439.268671:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078439.974800:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078440.292716:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078440.997376:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078441.316697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078442.020843:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078442.341025:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078443.046618:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078443.365524:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078444.068808:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078444.069165:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078444.069175:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078444.069180:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078444.069189:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5133 00000100:00080000:3.0:1769078444.069200:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078444.069207:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078444.069241:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923987968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078444.069281:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078444.069285:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078444.069287:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078444.069292:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5133 00000100:00080000:3.0:1769078444.069295:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078444.069301:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078444.069311:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078444.069324:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923988096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078444.069336:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078444.069342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923987968:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078444.069343:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078444.069347:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078444.069359:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5133 00000100:00080000:3.0:1769078444.069378:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078444.069382:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078444.069388:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078444.069401:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923988224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078444.069410:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078444.069414:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078444.069415:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078444.069420:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5133 00000100:00080000:3.0:1769078444.069427:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078444.069431:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078444.069435:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078444.069440:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078444.069453:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923988352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078444.069465:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078444.069468:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078444.069471:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078444.069477:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5133 00000100:00080000:3.0:1769078444.069481:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078444.069487:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078444.069492:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078444/real 1769078444] req@ffff969e0a7ffb80 x1855007923987968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078444.069495:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078444.069509:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923988480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078444.069553:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923987968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078444.069573:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078444.069579:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078444.069585:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923987968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078444.070738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078444.070900:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078444.070916:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078444/real 1769078444] req@ffff969e0a7fd880 x1855007923988096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078444.070931:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923988096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078444.070940:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078444.070943:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078444.070946:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988096:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078444.070954:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078444.070973:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078444.070984:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078444/real 1769078444] req@ffff969e0a7fd180 x1855007923988224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078444.070996:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923988224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078444.071004:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078444.071007:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078444.071010:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078444.071017:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078444.071036:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078444.071047:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078444/real 1769078444] req@ffff969e0a7fe300 x1855007923988352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078444.071059:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923988352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078444.071066:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078444.071070:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078444.071073:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988352:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078444.071078:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078444.071096:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078444.071107:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078444/real 1769078444] req@ffff969e0a7fca80 x1855007923988480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078444.071117:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923988480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078444.071125:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078444.071128:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078444.071131:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078444.388731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078445.093463:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078445.412739:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078446.116975:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078446.436724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078447.140781:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078447.464328:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078448.164813:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078448.484713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078449.188771:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078449.188784:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078449.188788:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078449.188799:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5138 00000100:00080000:0.0:1769078449.188811:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078449.188818:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078449.188857:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923988608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078449.188883:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078449.188887:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078449.188889:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078449.188898:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5138 00000100:00080000:0.0:1769078449.188902:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078449.188906:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078449.188915:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078449.188931:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923988608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078449.188949:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923988736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078449.188984:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078449.188988:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078449.188990:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078449.188995:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5138 00000100:00080000:0.0:1769078449.188999:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1769078449.189004:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078449.189007:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078449.189025:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078449.189045:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078449/real 1769078449] req@ffff969e06a60000 x1855007923988608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078449.189064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923988608/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078449.189074:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078449.189078:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078449.189081:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923988608:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078449.189104:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988736:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078449.189125:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078449.189137:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078449/real 1769078449] req@ffff969e06a63800 x1855007923988736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078449.189148:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923988736/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078449.189156:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078449.189159:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078449.189190:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988736:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078449.189270:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923988864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078449.189291:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078449.189295:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078449.189300:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078449.189303:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078449.189307:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5138 00000100:00080000:0.0:1769078449.189312:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078449.189317:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078449.189333:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078449.189337:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:0.0:1769078449.189348:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923988992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078449.189360:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078449.189365:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078449.189367:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078449.189371:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078449.189376:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5138 00000100:00100000:1.0:1769078449.189380:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078449/real 1769078449] req@ffff969e06a60380 x1855007923988864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078449.189389:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078449.189398:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078449.189401:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923988864/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078449.189405:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078449.189416:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078449.189420:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078449.189424:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923989120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078449.189428:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988864:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078449.189450:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988992:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078449.189472:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078449.189483:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078449/real 1769078449] req@ffff969e06a60a80 x1855007923988992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078449.189495:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923988992/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078449.189503:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078449.189506:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078449.189510:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923988992:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078449.189515:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989120:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078449.189533:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078449.189543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078449/real 1769078449] req@ffff969e06a61500 x1855007923989120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078449.189555:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923989120/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078449.189569:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078449.189572:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078449.189576:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989120:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078449.508753:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078450.213301:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078450.532892:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078451.238453:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078451.556685:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078452.260826:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078452.580757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078453.284874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078453.605173:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078454.308835:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078454.311130:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078454.311138:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078454.311143:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078454.311154:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5143 00000100:00080000:3.0:1769078454.311166:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078454.311172:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078454.311218:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923989248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078454.311248:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078454.311252:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078454.311254:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078454.311258:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5143 00000100:00080000:3.0:1769078454.311262:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078454.311267:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078454.311277:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078454.311286:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923989248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078454.311290:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923989376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078454.311298:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078454.311301:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078454.311302:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078454.311306:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5143 00000100:00080000:3.0:1769078454.311308:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078454.311312:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078454.311334:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078454.311344:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923989504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078454.311352:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078454.311355:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078454.311355:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078454.311357:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078454.311360:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5143 00000100:00080000:3.0:1769078454.311363:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078454.311366:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078454.311370:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078454.311381:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923989632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078454.311388:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078454.311391:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078454.311393:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078454.311396:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5143 00000100:00080000:3.0:1769078454.311399:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078454.311402:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078454/real 1769078454] req@ffff969e0a7fdf80 x1855007923989248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078454.311402:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078454.311407:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078454.311415:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923989760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078454.311420:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923989248/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078454.311430:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078454.311434:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078454.311437:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923989248:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078454.311462:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989376:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078454.311495:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078454.311508:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078454/real 1769078454] req@ffff969e0a7fea00 x1855007923989376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078454.311519:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923989376/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078454.311528:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078454.311532:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078454.311535:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989376:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078454.311541:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989504:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078454.311560:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078454.311571:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078454/real 1769078454] req@ffff969e0a7fc700 x1855007923989504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078454.311582:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923989504/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078454.311607:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078454.311611:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078454.311615:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989504:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078454.311620:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989632:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078454.311640:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078454.311651:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078454/real 1769078454] req@ffff969e0a7fce00 x1855007923989632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078454.311663:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923989632/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078454.311671:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078454.311674:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078454.311678:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989632:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078454.311684:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989760:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078454.311701:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078454.311710:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078454/real 1769078454] req@ffff969e0a7fe680 x1855007923989760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078454.311722:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923989760/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078454.311730:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078454.311732:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078454.311736:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923989760:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078454.628678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078455.332796:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078455.652730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078456.356851:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078456.676688:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078457.382490:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078457.700709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078458.406950:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078458.724746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078459.429215:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078459.429230:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078459.429236:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078459.429250:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5148 00000100:00080000:0.0:1769078459.429262:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078459.429271:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078459.429330:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923989888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078459.429363:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078459.429366:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078459.429368:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078459.429372:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5148 00000100:00080000:0.0:1769078459.429376:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078459.429380:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078459.429392:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078459.429404:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923990016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078459.429413:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078459.429748:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078459.429750:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078459.429756:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5148 00000100:00080000:0.0:1769078459.429760:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078459.429764:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078459.429770:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078459.429786:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923990144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078459.429795:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078459.429799:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078459.429801:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078459.429805:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5148 00000100:00080000:0.0:1769078459.429809:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078459.429813:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078459.429817:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078459.429828:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923990272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078459.429835:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078459.429838:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078459.429840:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078459.429843:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5148 00000100:00080000:0.0:1769078459.429846:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078459.429850:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078459.429860:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078459.429873:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923990400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078459.430407:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923989888:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078459.430493:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078459.430531:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078459/real 1769078459] req@ffff969e06a60700 x1855007923989888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078459.430551:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923989888/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078459.430561:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078459.430564:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078459.430569:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923989888:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078459.430576:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990016:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078459.430620:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078459.430633:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078459/real 1769078459] req@ffff969e06a62d80 x1855007923990016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078459.430645:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923990016/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078459.430653:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078459.430656:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078459.430659:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990016:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078459.430665:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990144:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078459.430684:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078459.430703:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078459/real 1769078459] req@ffff969e06a62680 x1855007923990144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078459.430717:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923990144/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078459.430724:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078459.430728:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078459.430731:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990144:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078459.430738:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990272:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078459.430759:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078459.430770:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078459/real 1769078459] req@ffff969e06a61880 x1855007923990272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078459.430780:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923990272/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078459.430788:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078459.430791:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078459.430794:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990272:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078459.430799:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990400:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078459.430816:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078459.430825:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078459/real 1769078459] req@ffff969e06a61180 x1855007923990400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078459.430834:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923990400/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078459.430842:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078459.430845:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078459.430847:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990400:192.168.204.143@tcp:8:kworker.0 00010000:00010000:2.0:1769078459.436860:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078459.748760:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078460.452842:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078460.774793:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078461.477940:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078461.797171:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078462.500806:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078462.821684:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078463.526239:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078463.844759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078464.548766:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078464.548779:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078464.548784:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078464.548809:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5153 00000100:00080000:0.0:1769078464.548822:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078464.548831:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078464.548851:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078464.548959:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923990528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078464.549025:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923990528:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078464.549101:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078464.549135:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078464.549139:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078464.549141:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078464.549146:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5153 00000100:00080000:0.0:1769078464.549150:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078464.549154:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078464.549165:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078464.549180:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923990656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078464.549237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078464/real 1769078464] req@ffff969e06a63100 x1855007923990528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078464.549257:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923990528/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078464.549266:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078464.549270:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078464.549274:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923990528:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078464.549301:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990656:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078464.549323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078464.549335:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078464/real 1769078464] req@ffff969e06a61f80 x1855007923990656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078464.549347:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923990656/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078464.549354:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078464.549357:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078464.549360:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990656:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078464.549409:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078464.549417:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078464.549419:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078464.549423:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5153 00000100:00080000:0.0:1769078464.549446:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078464.549451:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078464.549457:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078464.549470:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007923990784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078464.549497:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078464.549501:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078464.549503:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078464.549508:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5153 00000100:00080000:0.0:1769078464.549512:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078464.549516:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078464.549520:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990784:192.168.204.143@tcp:38:kworker.0 00000008:00100000:0.0:1769078464.549524:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078464.549540:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007923990912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078464.549551:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1769078464.549553:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078464.549558:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078464.549561:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078464.549567:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5153 00000100:00080000:0.0:1769078464.549572:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078464.549572:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078464/real 1769078464] req@ffff969e06a61f80 x1855007923990784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078464.549581:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078464.549617:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007923990784/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769078464.549622:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078464.549627:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078464.549632:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078464.549638:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923991040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078464.549639:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990784:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078464.549658:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990912:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078464.549687:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078464.549699:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078464/real 1769078464] req@ffff969e06a63100 x1855007923990912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078464.549709:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007923990912/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078464.549718:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078464.549721:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078464.549724:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923990912:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078464.549734:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991040:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078464.549750:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078464.549758:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078464/real 1769078464] req@ffff969e06a61180 x1855007923991040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078464.549769:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923991040/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078464.549782:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078464.549785:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078464.549788:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991040:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078464.868707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078465.572916:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078465.892736:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078466.596823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078466.916940:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078467.620839:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078467.940734:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078468.644972:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078468.964703:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078469.668788:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078469.668799:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078469.668804:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078469.668814:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5158 00000100:00080000:3.0:1769078469.668835:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078469.668847:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078469.668877:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923991168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078469.668898:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078469.668901:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078469.668902:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078469.668905:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5158 00000100:00080000:3.0:1769078469.668907:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078469.668909:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078469.668916:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078469.668924:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923991296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078469.668930:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078469.668932:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078469.668933:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078469.668935:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5158 00000100:00080000:3.0:1769078469.668948:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078469.668951:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078469.668954:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078469.668961:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923991424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078469.668967:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078469.668969:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078469.668970:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078469.668973:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5158 00000100:00080000:3.0:1769078469.668975:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078469.668978:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078469.668982:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078469.668990:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923991552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078469.668999:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078469.669002:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078469.669004:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078469.669007:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5158 00000100:00080000:3.0:1769078469.669010:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078469.669013:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078469.669018:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078469.669027:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923991680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1769078469.671444:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078469.671727:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923991168:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078469.671792:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078469.671843:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078469/real 1769078469] req@ffff969e0a7fed80 x1855007923991168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078469.671858:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923991168/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078469.671866:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078469.671869:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078469.671871:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923991168:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078469.671877:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991296:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078469.671894:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078469.671912:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078469/real 1769078469] req@ffff969e0a7fe680 x1855007923991296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078469.671936:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923991296/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078469.671946:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078469.671949:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078469.671953:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991296:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078469.671960:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991424:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078469.671981:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078469.671991:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078469/real 1769078469] req@ffff969e0a7fce00 x1855007923991424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078469.671999:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923991424/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078469.672005:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078469.672008:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078469.672011:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991424:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078469.672018:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991552:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078469.672037:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078469.672048:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078469/real 1769078469] req@ffff969e0a7fc700 x1855007923991552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078469.672059:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923991552/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078469.672068:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078469.672072:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078469.672075:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991552:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078469.672082:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991680:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078469.672099:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078469.672108:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078469/real 1769078469] req@ffff969e0a7fea00 x1855007923991680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078469.672120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923991680/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078469.672140:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078469.672143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078469.672147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991680:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078469.990944:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078470.692934:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078471.015084:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078471.718751:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078472.036711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078472.742381:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078473.060708:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078473.764778:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078474.084678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078474.788749:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078474.788760:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078474.788764:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078474.788777:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5163 00000100:00080000:3.0:1769078474.788789:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078474.788797:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078474.788819:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078474.788843:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923991808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078474.788874:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078474.788877:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078474.788880:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078474.788884:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5163 00000100:00080000:3.0:1769078474.788888:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078474.788891:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078474.788901:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078474.788915:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923991936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078474.788920:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923991808:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078474.788926:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078474.788940:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078474.788961:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078474.788965:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5163 00000100:00080000:3.0:1769078474.788969:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078474.788972:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078474.788976:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078474.788987:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923992064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078474.788994:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078474.788997:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078474.788998:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078474.789000:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5163 00000100:00080000:3.0:1769078474.789003:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078474.789005:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078474.789008:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000800:00000100:1.0:1769078474.789011:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078474.789022:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923992192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078474.789030:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078474.789033:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078474.789035:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078474.789038:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5163 00000100:00080000:3.0:1769078474.789042:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078474.789045:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078474.789048:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078474.789058:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078474/real 1769078474] req@ffff969e0a7fdf80 x1855007923991808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078474.789062:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923992320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078474.789133:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923991808/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078474.789146:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078474.789150:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078474.789154:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923991808:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078474.789179:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991936:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078474.789200:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078474.789211:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078474/real 1769078474] req@ffff969e0a7fca80 x1855007923991936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078474.789223:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923991936/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078474.789238:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078474.789241:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078474.789245:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923991936:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078474.789251:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992064:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078474.789268:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078474.789277:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078474/real 1769078474] req@ffff969e0a7fe300 x1855007923992064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078474.789288:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923992064/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078474.789296:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078474.789299:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078474.789302:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992064:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078474.789308:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992192:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078474.789323:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078474.789331:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078474/real 1769078474] req@ffff969e0a7fd180 x1855007923992192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078474.789342:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923992192/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078474.789349:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078474.789352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078474.789355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992192:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078474.789361:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992320:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078474.789375:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078474.789383:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078474/real 1769078474] req@ffff969e0a7fd880 x1855007923992320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078474.789394:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923992320/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078474.789402:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078474.789405:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078474.789408:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992320:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078475.108707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078475.812821:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078476.132723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078476.836807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078477.156725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078477.860763:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078478.180749:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078478.884792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078479.204718:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078479.908931:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078479.910916:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078479.910925:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078479.910929:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078479.910940:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5169 00000100:00080000:3.0:1769078479.910950:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078479.910956:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078479.910991:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923992448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078479.911022:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078479.911026:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078479.911028:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078479.911032:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5169 00000100:00080000:3.0:1769078479.911036:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078479.911040:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078479.911049:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078479.911061:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923992576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078479.911070:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078479.911085:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078479.911087:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078479.911091:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5169 00000100:00080000:3.0:1769078479.911095:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078479.911099:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078479.911104:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078479.911112:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923992704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078479.911118:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078479.911120:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078479.911122:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078479.911124:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5169 00000100:00080000:3.0:1769078479.911126:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078479.911128:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078479.911130:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078479.911137:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923992832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078479.911141:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078479.911143:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078479.911144:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078479.911146:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5169 00000100:00080000:3.0:1769078479.911148:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078479.911150:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078479.911156:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078479.911163:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923992960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078479.912132:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923992448:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078479.912208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078479.912246:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078479/real 1769078479] req@ffff969e0a7ffb80 x1855007923992448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078479.912262:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923992448/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078479.912272:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078479.912276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078479.912280:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923992448:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078479.912288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992576:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078479.912308:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078479.912317:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078479/real 1769078479] req@ffff969e0a7ff800 x1855007923992576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078479.912324:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923992576/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078479.912329:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078479.912331:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078479.912334:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992576:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078479.912337:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992704:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078479.912348:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078479.912354:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078479/real 1769078479] req@ffff969e0a7fdc00 x1855007923992704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078479.912364:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923992704/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078479.912370:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078479.912373:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078479.912376:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992704:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078479.912381:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992832:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078479.912399:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078479.912408:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078479/real 1769078479] req@ffff969e0a7fc000 x1855007923992832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078479.912418:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923992832/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078479.912425:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078479.912428:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078479.912431:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992832:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078479.912437:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992960:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078479.912465:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078479.912476:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078479/real 1769078479] req@ffff969e0a7fd500 x1855007923992960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078479.912485:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923992960/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078479.912492:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078479.912495:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078479.912497:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923992960:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078480.228742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078480.932797:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078481.254803:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078481.956737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078482.276780:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078482.982042:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078483.300759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078484.005962:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078484.326403:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078485.028711:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078485.028723:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078485.028729:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078485.028741:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5174 00000100:00080000:3.0:1769078485.028766:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078485.028774:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:1.0:1769078485.028792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078485.028840:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007923993088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078485.029288:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078485.029292:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078485.029294:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078485.029307:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5174 00000100:00080000:3.0:1769078485.029312:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078485.029317:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078485.029328:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078485.029345:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007923993216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078485.029378:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078485.029381:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078485.029395:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078485.029400:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5174 00000100:00080000:3.0:1769078485.029404:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078485.029409:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923993088:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078485.029422:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078485.029429:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078485.029451:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007923993344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078485.029461:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078485.029464:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078485.029477:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078485.029481:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5174 00000100:00080000:3.0:1769078485.029485:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078485.029489:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078485.029494:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078485.029514:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007923993472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078485.029530:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078485.029534:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078485.029535:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078485.029539:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5174 00000100:00080000:3.0:1769078485.029555:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078485.029587:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769078485.029944:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078485.029983:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078485/real 1769078485] req@ffff969e0a7fc380 x1855007923993088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078485.030002:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007923993088/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078485.030013:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078485.030017:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078485.030021:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923993088:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078485.030041:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993216:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078485.030064:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078485.030076:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078485/real 1769078485] req@ffff969e0a7fd500 x1855007923993216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078485.030100:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007923993216/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078485.030115:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078485.030119:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078485.030122:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993216:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078485.030149:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993344:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078485.030169:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078485.030179:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078485/real 1769078485] req@ffff969e0a7fc000 x1855007923993344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078485.030190:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007923993344/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078485.030199:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078485.030202:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078485.030205:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993344:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078485.030222:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993472:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078485.030241:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078485.030252:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078485/real 1769078485] req@ffff969e0a7fdc00 x1855007923993472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078485.030262:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007923993472/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078485.030270:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078485.030273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078485.030276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993472:192.168.204.143@tcp:8:kworker.0 00000008:00100000:3.0:1769078485.031360:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078485.031389:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007923993600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078485.031466:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993600:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078485.031502:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078485.031516:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078485/real 1769078485] req@ffff969e0a7ff800 x1855007923993600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078485.031527:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007923993600/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078485.031534:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078485.031536:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078485.031538:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993600:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078485.351955:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078486.053893:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078486.372744:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078487.076785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078487.396732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078488.100707:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078488.421197:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078489.124725:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078489.444746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078490.148812:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1769078490.148823:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078490.148826:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078490.148833:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078490.148845:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5179 00000100:00080000:0.0:1769078490.148858:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078490.148866:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078490.148914:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923993728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078490.149037:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923993728:192.168.204.143@tcp:250:kworker.0 00000800:00000100:2.0:1769078490.149182:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078490.149212:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078490.149217:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078490.149219:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078490.149224:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5179 00000100:00080000:0.0:1769078490.149228:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078490.149232:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078490.149313:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078490.149330:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923993856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078490.149384:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078490/real 1769078490] req@ffff969e06a61880 x1855007923993728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078490.149402:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923993728/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078490.149412:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078490.149416:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078490.149421:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923993728:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078490.149448:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993856:192.168.204.143@tcp:38:kworker.0 00000100:00080000:0.0:1769078490.149494:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078490.149498:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078490.149500:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078490.149504:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5179 00000100:00080000:0.0:1769078490.149514:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078490.149518:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078490.149523:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078490.149534:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923993984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078490.149544:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078490.149548:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078490.149550:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078490.149553:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5179 00000100:00080000:0.0:1769078490.149557:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078490.149560:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078490.149565:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078490.149574:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923994112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078490.149583:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078490.149587:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078490.149588:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078490.149720:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5179 00000100:00080000:0.0:1769078490.149724:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769078490.149886:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078490.149901:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078490/real 1769078490] req@ffff969e06a62680 x1855007923993856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078490.149914:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923993856/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078490.149923:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078490.149927:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078490.149930:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993856:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078490.149944:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993984:192.168.204.143@tcp:38:kworker.0 00000800:00000100:2.0:1769078490.149964:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078490.149974:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078490/real 1769078490] req@ffff969e06a62d80 x1855007923993984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078490.149984:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923993984/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078490.149992:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078490.149996:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078490.149999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923993984:192.168.204.143@tcp:38:kworker.0 00000100:00100000:2.0:1769078490.150005:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994112:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078490.150023:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078490.150032:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078490/real 1769078490] req@ffff969e06a60700 x1855007923994112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078490.150056:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923994112/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078490.150085:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078490.150091:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078490.150107:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923994240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078490.150282:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078490.150286:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078490.150289:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994112:192.168.204.143@tcp:8:kworker.0 00000100:00100000:2.0:1769078490.150304:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994240:192.168.204.143@tcp:8:kworker.0 00000800:00000100:2.0:1769078490.150326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:2.0:1769078490.150337:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078490/real 1769078490] req@ffff969e06a61500 x1855007923994240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078490.150348:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923994240/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078490.150356:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078490.150359:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:2.0:1769078490.150362:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994240:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078490.468750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078491.172756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078491.492682:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078492.196708:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078492.516729:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078493.221670:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078493.540754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078494.244718:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078494.564752:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078495.269351:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078495.269364:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078495.269369:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078495.269381:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5184 00000100:00080000:2.0:1769078495.269403:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078495.269412:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078495.269460:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923994368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078495.269518:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078495.269522:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078495.269524:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078495.269529:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5184 00000100:00080000:2.0:1769078495.269533:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078495.269537:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078495.269549:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078495.269562:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923994496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078495.269572:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078495.269575:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078495.269577:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078495.269581:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5184 00000100:00080000:2.0:1769078495.269584:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078495.269588:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078495.269788:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078495.269802:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923994624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078495.269812:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078495.269816:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078495.269817:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078495.269821:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5184 00000100:00080000:2.0:1769078495.269831:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078495.269835:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078495.269840:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078495.269851:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923994752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078495.269860:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078495.269863:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078495.269865:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078495.269868:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5184 00000100:00080000:2.0:1769078495.269872:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078495.269875:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078495.269879:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078495.269888:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923994880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078495.271527:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923994368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078495.271648:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078495.271679:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078495.271699:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078495.271716:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994880:192.168.204.143@tcp:8:kworker.0 00010000:00010000:1.0:1769078495.271788:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078495.588756:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078496.294838:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078496.612786:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078497.316874:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078497.636770:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078498.340894:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769078498.341840:0:2407:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:0.0:1769078498.341897:0:2407:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769078498.341919:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078498.341968:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078498.341979:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078498.341987:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:0.0:1769078498.341995:0:2407:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769078498.342122:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078495/real 1769078498] req@ffff969e0a46ea00 x1855007923994368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078498.342150:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923994368/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078498.342161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078498.342166:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078498.342171:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923994368:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078498.342178:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078495/real 1769078498] req@ffff969e0a46d500 x1855007923994496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078498.342196:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923994496/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078498.342203:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078498.342206:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078498.342209:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994496:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078498.342215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078495/real 1769078498] req@ffff969e0a46ca80 x1855007923994624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078498.342225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923994624/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078498.342232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078498.342235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078498.342238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994624:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078498.342244:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078495/real 1769078498] req@ffff969e0a46df80 x1855007923994752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078498.342256:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923994752/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078498.342263:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078498.342273:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078498.342276:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994752:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078498.342293:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078495/real 1769078498] req@ffff969e0a46f100 x1855007923994880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078498.342307:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923994880/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078498.342315:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078498.342318:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078498.342322:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923994880:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078498.662730:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078499.364738:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078499.684699:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078500.388714:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078500.388726:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078500.388730:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078500.388740:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5189 00000100:00080000:2.0:1769078500.388752:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078500.388760:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:1.0:1769078500.388766:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078500.388792:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46e300 x1855007923995008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078500.388820:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078500.388838:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078500.388840:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078500.388845:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5189 00000100:00080000:2.0:1769078500.388849:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078500.388852:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078500.388861:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078500.388873:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ed80 x1855007923995136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078500.388883:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078500.388886:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078500.388888:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078500.388892:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5189 00000100:00080000:2.0:1769078500.388896:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078500.388899:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078500.388904:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078500.388905:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923995008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:2.0:1769078500.388917:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f100 x1855007923995264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078500.388928:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078500.388932:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078500.388934:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078500.388937:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5189 00000100:00080000:2.0:1769078500.388941:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078500.388945:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078500.388954:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078500.388964:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46df80 x1855007923995392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078500.388975:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078500.388977:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078500.388981:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078500.388993:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078500.388999:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5189 00000100:00080000:2.0:1769078500.389003:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078500.389008:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078500.389013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078500/real 1769078500] req@ffff969e0a46e300 x1855007923995008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1769078500.389015:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078500.389028:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ca80 x1855007923995520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078500.389053:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46e300 x1855007923995008/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078500.389064:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078500.389076:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078500.389080:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923995008:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078500.389088:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995136:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078500.389112:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078500.389124:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078500/real 1769078500] req@ffff969e0a46ed80 x1855007923995136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078500.389136:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ed80 x1855007923995136/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078500.389144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078500.389147:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078500.389160:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995136:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078500.389187:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995264:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078500.389208:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078500.389220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078500/real 1769078500] req@ffff969e0a46f100 x1855007923995264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078500.389233:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f100 x1855007923995264/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078500.389240:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078500.389243:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078500.389246:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995264:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078500.389252:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995392:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078500.389274:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078500.389286:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078500/real 1769078500] req@ffff969e0a46df80 x1855007923995392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078500.389297:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46df80 x1855007923995392/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078500.389336:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078500.389340:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078500.389345:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995392:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078500.389351:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995520:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078500.389374:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078500.389392:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078500/real 1769078500] req@ffff969e0a46ca80 x1855007923995520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078500.389404:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ca80 x1855007923995520/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078500.389411:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078500.389415:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078500.389418:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995520:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078500.708790:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078501.412754:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078501.732797:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078502.436976:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078502.756697:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078503.460773:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078503.780741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078504.486380:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078504.804722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078505.508772:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078505.508788:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078505.509066:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078505.509080:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5194 00000100:00080000:2.0:1769078505.509093:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078505.509101:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078505.509149:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d500 x1855007923995648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078505.509180:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078505.509184:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078505.509186:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078505.509189:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5194 00000100:00080000:2.0:1769078505.509232:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078505.509237:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078505.509249:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078505.509263:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ea00 x1855007923995776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078505.509273:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078505.509276:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078505.509278:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078505.509282:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5194 00000100:00080000:2.0:1769078505.509285:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078505.509289:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078505.509306:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078505.509319:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46c380 x1855007923995904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078505.509328:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078505.509332:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078505.509334:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078505.509337:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5194 00000100:00080000:2.0:1769078505.509340:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078505.509344:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078505.509348:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078505.509359:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46ce00 x1855007923996032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078505.509381:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078505.509393:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078505.509395:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078505.509400:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5194 00000100:00080000:2.0:1769078505.509404:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078505.509407:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078505.509412:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078505.509422:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007923996160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078505.509630:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923995648:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078505.509713:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078505.510157:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078505/real 1769078505] req@ffff969e0a46d500 x1855007923995648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078505.510174:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d500 x1855007923995648/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078505.510184:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078505.510188:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078505.510201:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923995648:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078505.510209:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995776:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078505.510242:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078505.510255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078505/real 1769078505] req@ffff969e0a46ea00 x1855007923995776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078505.510268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ea00 x1855007923995776/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078505.510275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078505.510278:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078505.510282:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995776:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078505.510288:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995904:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078505.510304:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078505.510313:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078505/real 1769078505] req@ffff969e0a46c380 x1855007923995904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078505.510323:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46c380 x1855007923995904/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078505.510331:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078505.510335:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078505.510337:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923995904:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078505.510342:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996032:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078505.510357:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078505.510365:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078505/real 1769078505] req@ffff969e0a46ce00 x1855007923996032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078505.510376:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46ce00 x1855007923996032/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078505.510385:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078505.510398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078505.510401:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996032:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078505.510406:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996160:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078505.510422:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078505.510432:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078505/real 1769078505] req@ffff969e0a46f800 x1855007923996160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078505.510441:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007923996160/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078505.510450:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078505.510454:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078505.510457:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996160:192.168.204.143@tcp:8:kworker.0 00010000:00010000:1.0:1769078505.511050:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078505.830486:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078506.532762:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078506.852733:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078507.557498:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078507.876707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078508.582327:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078508.900707:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078509.604712:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078509.924726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078510.628736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078510.628742:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078510.628749:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078510.628753:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078510.628763:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5199 00000100:00080000:3.0:1769078510.628772:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078510.628778:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078510.628818:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007923996288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078510.628839:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078510.628842:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078510.628843:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078510.628846:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5199 00000100:00080000:3.0:1769078510.628848:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078510.628851:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078510.628858:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078510.628870:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007923996416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078510.628871:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923996288:192.168.204.143@tcp:250:kworker.0 00000100:00080000:3.0:1769078510.628883:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078510.628885:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078510.628886:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078510.628888:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5199 00000100:00080000:3.0:1769078510.628890:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078510.628893:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078510.628897:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078510.628929:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923996544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078510.628938:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078510.628942:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078510.628946:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078510.628947:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078510.628954:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5199 00000100:00080000:3.0:1769078510.628959:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078510.628963:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078510.628968:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078510.628978:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923996672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078510.628985:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1769078510.628986:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078510/real 1769078510] req@ffff969e0a7ffb80 x1855007923996288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078510.628992:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078510.628996:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078510.629002:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5199 00000100:00080000:3.0:1769078510.629005:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078510.629007:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078510.629010:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078510.629012:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007923996288/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078510.629020:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923996800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078510.629024:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078510.629032:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078510.629036:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923996288:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078510.629056:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996416:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078510.629071:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078510.629079:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078510/real 1769078510] req@ffff969e0a7fd880 x1855007923996416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078510.629086:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007923996416/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078510.629092:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078510.629094:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078510.629096:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996416:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078510.629099:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996544:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078510.629108:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078510.629113:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078510/real 1769078510] req@ffff969e0a7fd180 x1855007923996544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078510.629120:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923996544/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078510.629153:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078510.629155:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078510.629157:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996544:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078510.629162:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996672:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078510.629181:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078510.629191:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078510/real 1769078510] req@ffff969e0a7fe300 x1855007923996672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078510.629202:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923996672/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078510.629210:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078510.629214:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078510.629216:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996672:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078510.629220:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996800:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078510.629234:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078510.629240:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078510/real 1769078510] req@ffff969e0a7fca80 x1855007923996800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078510.629247:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923996800/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078510.629252:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078510.629254:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078510.629256:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923996800:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078510.948653:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078511.652685:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078511.972723:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078512.676753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078512.996725:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078513.700744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078514.020740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078514.724759:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078515.044717:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078515.748738:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078515.748751:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1769078515.748752:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078515.748755:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078515.748766:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5204 00000100:00080000:3.0:1769078515.748778:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078515.748785:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078515.748815:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923996928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078515.748838:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078515.748846:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078515.748848:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078515.748851:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5204 00000100:00080000:3.0:1769078515.748855:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078515.748858:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078515.748865:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078515.748873:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923996928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078515.748875:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923997056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078515.748890:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078515.748893:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078515.748895:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078515.748899:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5204 00000100:00080000:3.0:1769078515.748902:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078515.748906:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078515.748927:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769078515.748935:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:3.0:1769078515.748942:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923997184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078515.748954:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078515.748957:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078515.748959:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078515.748966:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078515/real 1769078515] req@ffff969e0a7fdf80 x1855007923996928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078515.748966:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5204 00000100:00080000:3.0:1769078515.748972:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078515.748979:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078515.748983:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923996928/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1769078515.748986:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078515.748993:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078515.748997:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078515.748999:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923996928:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078515.749004:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923997312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078515.749013:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078515.749016:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078515.749018:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078515.749024:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997056:192.168.204.143@tcp:38:kworker.0 00000100:00080000:3.0:1769078515.749024:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5204 00000100:00080000:3.0:1769078515.749029:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078515.749034:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078515.749039:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078515.749068:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923997440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078515.749083:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078515.749095:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078515/real 1769078515] req@ffff969e0a7fea00 x1855007923997056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078515.749103:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923997056/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078515.749108:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078515.749111:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078515.749113:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997056:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078515.749117:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997184:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078515.749128:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078515.749133:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078515/real 1769078515] req@ffff969e0a7fc700 x1855007923997184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078515.749139:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923997184/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078515.749144:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078515.749145:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078515.749147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997184:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078515.749150:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997312:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078515.749159:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078515.749164:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078515/real 1769078515] req@ffff969e0a7fce00 x1855007923997312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078515.749170:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923997312/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078515.749175:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078515.749176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078515.749178:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997312:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078515.749201:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997440:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078515.749221:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078515.749232:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078515/real 1769078515] req@ffff969e0a7fe680 x1855007923997440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078515.749241:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923997440/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078515.749249:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078515.749251:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078515.749254:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997440:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078516.068731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078516.772758:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078517.092731:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078517.796724:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078518.116705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078518.820737:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078519.141397:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078519.845154:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078520.164719:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078520.868889:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078520.871098:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078520.871105:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078520.871108:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078520.871116:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5210 00000100:00080000:3.0:1769078520.871126:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078520.871132:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078520.871162:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fed80 x1855007923997568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078520.871189:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078520.871192:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078520.871193:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078520.871196:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5210 00000100:00080000:3.0:1769078520.871199:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078520.871201:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078520.871209:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078520.871218:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe680 x1855007923997696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078520.871225:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078520.871227:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078520.871228:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078520.871230:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5210 00000100:00080000:3.0:1769078520.871232:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078520.871246:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078520.871250:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078520.871256:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fce00 x1855007923997824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078520.871263:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078520.871265:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078520.871266:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078520.871268:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5210 00000100:00080000:3.0:1769078520.871270:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078520.871272:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078520.871274:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078520.871279:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc700 x1855007923997952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078520.871284:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078520.871286:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078520.871287:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078520.871288:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5210 00000100:00080000:3.0:1769078520.871290:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078520.871291:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078520.871293:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078520.871298:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fea00 x1855007923998080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078520.871474:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923997568:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078520.871529:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078520.871580:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078520/real 1769078520] req@ffff969e0a7fed80 x1855007923997568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078520.871615:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fed80 x1855007923997568/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078520.871871:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078520.871877:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078520.871881:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923997568:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078520.871889:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997696:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078520.871912:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078520.871925:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078520/real 1769078520] req@ffff969e0a7fe680 x1855007923997696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078520.871943:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe680 x1855007923997696/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078520.871950:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078520.871953:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078520.871983:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997696:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078520.871989:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997824:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078520.872007:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078520.872017:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078520/real 1769078520] req@ffff969e0a7fce00 x1855007923997824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078520.872027:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fce00 x1855007923997824/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078520.872033:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078520.872036:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078520.872039:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997824:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078520.872044:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997952:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078520.872062:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078520.872071:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078520/real 1769078520] req@ffff969e0a7fc700 x1855007923997952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078520.872081:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc700 x1855007923997952/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078520.872094:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078520.872098:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078520.872101:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923997952:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078520.872106:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998080:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078520.872123:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078520.872146:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078520/real 1769078520] req@ffff969e0a7fea00 x1855007923998080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078520.872155:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fea00 x1855007923998080/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078520.872161:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078520.872163:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078520.872164:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998080:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078521.190103:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078521.892820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078522.212670:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078522.916830:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078523.236751:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078523.941785:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078524.260722:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078524.964805:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078525.284746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078525.988820:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078525.989706:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078525.989713:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078525.989717:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078525.989728:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5215 00000100:00080000:0.0:1769078525.989739:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078525.989746:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078525.989777:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007923998208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078525.989808:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078525.989811:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078525.989814:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078525.989819:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5215 00000100:00080000:0.0:1769078525.989822:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078525.989827:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078525.989837:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078525.989847:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007923998336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078525.989866:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078525.989974:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078525.989976:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078525.989981:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5215 00000100:00080000:0.0:1769078525.989993:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078525.989997:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078525.990002:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078525.990013:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007923998464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078525.990022:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078525.990026:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078525.990027:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078525.990030:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5215 00000100:00080000:0.0:1769078525.990033:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078525.990036:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078525.990041:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078525.990049:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007923998592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078525.990058:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078525.990061:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078525.990063:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078525.990067:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5215 00000100:00080000:0.0:1769078525.990075:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078525.990079:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078525.990084:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078525.990094:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007923998720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078525.990279:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923998208:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078525.990342:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078525.990370:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078525/real 1769078525] req@ffff969e06a60a80 x1855007923998208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078525.990386:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007923998208/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078525.990395:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078525.990398:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078525.990402:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923998208:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078525.990409:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998336:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078525.990427:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078525.990439:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078525/real 1769078525] req@ffff969e06a60380 x1855007923998336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078525.990449:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007923998336/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078525.990456:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078525.990459:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078525.990462:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998336:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078525.990468:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998464:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078525.990483:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078525.990492:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078525/real 1769078525] req@ffff969e06a63800 x1855007923998464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078525.990501:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007923998464/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078525.990508:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078525.990511:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078525.990514:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998464:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078525.990519:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998592:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078525.990534:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078525.990543:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078525/real 1769078525] req@ffff969e06a60000 x1855007923998592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078525.990553:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007923998592/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078525.990561:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078525.990564:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078525.990566:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998592:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078525.990571:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998720:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078525.990584:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078525.990699:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078525/real 1769078525] req@ffff969e06a61500 x1855007923998720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078525.990711:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007923998720/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078525.990718:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078525.990722:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078525.990725:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998720:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078526.308713:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078527.013026:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078527.332678:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078528.043747:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078528.356747:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078529.061156:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078529.380878:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078530.087232:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078530.404728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078531.108970:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078531.109185:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078531.109195:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078531.109200:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078531.109209:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5220 00000100:00080000:0.0:1769078531.109221:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078531.109228:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078531.109262:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007923998848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078531.109288:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078531.109292:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078531.109294:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078531.109298:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5220 00000100:00080000:0.0:1769078531.109302:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078531.109306:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078531.109316:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078531.109329:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007923998976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078531.109338:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923998848:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078531.109345:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078531.109368:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078531.109371:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078531.109377:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5220 00000100:00080000:0.0:1769078531.109382:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078531.109386:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078531.109391:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078531.109403:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007923999104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078531.109412:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078531.109415:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078531.109417:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078531.109420:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5220 00000100:00080000:0.0:1769078531.109424:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078531.109426:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078531.109430:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078531.109438:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007923999232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078531.109445:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078531.109447:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078531.109449:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078531.109452:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5220 00000100:00080000:0.0:1769078531.109454:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078531.109458:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078531.109466:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078531.109475:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007923999360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078531.109690:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078531.109731:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078531/real 1769078531] req@ffff969e06a60700 x1855007923998848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078531.109750:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007923998848/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078531.109768:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078531.109773:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078531.109778:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923998848:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078531.109816:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998976:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078531.109837:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078531.109849:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078531/real 1769078531] req@ffff969e06a62d80 x1855007923998976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078531.109861:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007923998976/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078531.109870:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078531.109873:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078531.109876:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923998976:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078531.109883:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999104:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078531.109898:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078531.109906:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078531/real 1769078531] req@ffff969e06a62680 x1855007923999104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078531.109916:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007923999104/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078531.109924:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078531.109928:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078531.109931:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999104:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078531.109937:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999232:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078531.109952:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078531.109960:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078531/real 1769078531] req@ffff969e06a61880 x1855007923999232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078531.109971:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007923999232/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078531.109979:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078531.109983:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078531.109986:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999232:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078531.109992:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999360:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078531.110005:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078531.110013:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078531/real 1769078531] req@ffff969e06a61180 x1855007923999360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078531.110024:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007923999360/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078531.110032:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078531.110036:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078531.110038:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999360:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078531.428704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078532.134854:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078532.452895:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078533.156836:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078533.476783:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078534.182499:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078534.504810:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078535.205668:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078535.524757:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078536.229765:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078536.231404:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078536.231414:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078536.231418:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078536.231428:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5225 00000100:00080000:3.0:1769078536.231439:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078536.231446:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078536.231481:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdf80 x1855007923999488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078536.231514:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078536.231518:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078536.231520:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078536.231524:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5225 00000100:00080000:3.0:1769078536.231528:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078536.231532:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078536.231558:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078536.231571:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fca80 x1855007923999616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078536.231581:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078536.231584:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078536.231586:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078536.231613:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5225 00000100:00080000:3.0:1769078536.231617:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078536.231621:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078536.231625:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078536.231635:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fe300 x1855007923999744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078536.231644:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078536.231648:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078536.231649:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078536.231653:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5225 00000100:00080000:3.0:1769078536.231656:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078536.231659:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078536.231663:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078536.231672:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd180 x1855007923999872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078536.231680:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078536.231683:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078536.231689:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078536.231693:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5225 00000100:00080000:3.0:1769078536.231708:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078536.231712:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078536.231717:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078536.231727:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd880 x1855007924000000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078536.232014:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923999488:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078536.232077:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078536.232110:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078536/real 1769078536] req@ffff969e0a7fdf80 x1855007923999488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078536.232128:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdf80 x1855007923999488/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078536.232139:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078536.232143:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078536.232147:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007923999488:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078536.232153:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999616:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078536.232171:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078536.232181:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078536/real 1769078536] req@ffff969e0a7fca80 x1855007923999616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078536.232192:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fca80 x1855007923999616/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078536.232200:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078536.232203:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078536.232206:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999616:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078536.232211:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999744:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078536.232402:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078536.232414:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078536/real 1769078536] req@ffff969e0a7fe300 x1855007923999744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078536.232426:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fe300 x1855007923999744/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078536.232440:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078536.232443:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078536.232446:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999744:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078536.232453:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999872:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078536.232469:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078536.232477:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078536/real 1769078536] req@ffff969e0a7fd180 x1855007923999872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078536.232487:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd180 x1855007923999872/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078536.232496:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078536.232499:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078536.232502:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007923999872:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078536.232507:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000000:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078536.232521:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078536.232529:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078536/real 1769078536] req@ffff969e0a7fd880 x1855007924000000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078536.232540:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd880 x1855007924000000/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078536.232549:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078536.232551:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078536.232555:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000000:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078536.548759:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078537.252824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078537.572664:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078538.276792:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078538.596778:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078539.300882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078539.620704:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078540.324739:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078540.644710:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078541.348858:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078541.355150:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078541.355161:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078541.355165:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078541.355174:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5230 00000100:00080000:3.0:1769078541.355186:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078541.355192:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078541.355228:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ffb80 x1855007924000128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078541.355262:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078541.355265:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078541.355267:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078541.355271:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5230 00000100:00080000:3.0:1769078541.355275:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078541.355292:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078541.355300:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078541.355309:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007924000256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078541.355315:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078541.355317:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078541.355318:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078541.355321:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5230 00000100:00080000:3.0:1769078541.355324:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078541.355325:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078541.355328:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078541.355333:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007924000384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078541.355338:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078541.355340:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078541.355341:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078541.355343:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5230 00000100:00080000:3.0:1769078541.355345:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078541.355347:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078541.355349:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078541.355355:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007924000512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078541.355359:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078541.355364:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078541.355365:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078541.355367:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5230 00000100:00080000:3.0:1769078541.355369:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078541.355371:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078541.355373:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078541.355379:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007924000640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078541.355420:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924000128:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078541.355500:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078541.355535:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078541/real 1769078541] req@ffff969e0a7ffb80 x1855007924000128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078541.355554:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ffb80 x1855007924000128/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078541.355562:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078541.355566:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078541.355570:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924000128:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078541.355577:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000256:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078541.355630:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078541.355643:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078541/real 1769078541] req@ffff969e0a7ff800 x1855007924000256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078541.355655:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007924000256/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078541.355662:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078541.355665:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078541.355668:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000256:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078541.355673:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000384:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078541.355692:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078541.355702:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078541/real 1769078541] req@ffff969e0a7fdc00 x1855007924000384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078541.355714:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007924000384/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078541.355721:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078541.355724:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078541.355727:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000384:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078541.355733:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000512:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078541.355749:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078541.355757:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078541/real 1769078541] req@ffff969e0a7fc000 x1855007924000512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078541.355767:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007924000512/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078541.355774:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078541.355778:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078541.355781:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000512:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078541.355786:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000640:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078541.355801:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078541.355811:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078541/real 1769078541] req@ffff969e0a7fd500 x1855007924000640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078541.355821:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007924000640/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078541.355830:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078541.355833:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078541.355836:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000640:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078541.668726:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078542.372756:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078542.692705:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078543.396744:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078543.720698:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078544.420755:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078544.740689:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078545.446882:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078545.764732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078546.468941:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078546.470628:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078546.470642:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078546.470646:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078546.470660:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5235 00000100:00080000:0.0:1769078546.470672:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078546.470682:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078546.470739:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63100 x1855007924000768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078546.470769:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078546.470773:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078546.470775:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078546.470779:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5235 00000100:00080000:0.0:1769078546.470783:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078546.470787:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078546.470800:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078546.470816:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61f80 x1855007924000896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078546.470825:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078546.470829:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078546.470831:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078546.470845:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5235 00000100:00100000:1.0:1769078546.470858:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924000768:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078546.470874:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078546.470882:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078546.470892:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078546.470908:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007924001024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078546.470919:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078546.470923:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078546.470925:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078546.470931:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5235 00000100:00080000:0.0:1769078546.470935:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078546.470939:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078546.470944:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078546.470954:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007924001152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078546.470963:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078546.470967:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078546.470971:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078546.470973:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:0.0:1769078546.470983:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5235 00000100:00080000:0.0:1769078546.470990:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078546.470996:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078546.471002:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078546.471016:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007924001280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078546.471028:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078546/real 1769078546] req@ffff969e06a63100 x1855007924000768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078546.471064:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63100 x1855007924000768/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078546.471075:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078546.471080:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078546.471084:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924000768:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078546.471092:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000896:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078546.471116:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078546.471129:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078546/real 1769078546] req@ffff969e06a61f80 x1855007924000896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078546.471141:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61f80 x1855007924000896/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078546.471150:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078546.471153:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078546.471156:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924000896:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078546.471185:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001024:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078546.471205:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078546.471215:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078546/real 1769078546] req@ffff969e06a61180 x1855007924001024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078546.471225:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007924001024/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078546.471232:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078546.471235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078546.471238:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001024:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078546.471244:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001152:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078546.471260:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078546.471268:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078546/real 1769078546] req@ffff969e06a61880 x1855007924001152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078546.471278:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007924001152/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078546.471285:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078546.471289:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078546.471291:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001152:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078546.471809:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001280:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078546.471840:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078546.471860:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078546/real 1769078546] req@ffff969e06a62680 x1855007924001280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078546.471877:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007924001280/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078546.471886:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078546.471890:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078546.471893:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001280:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078546.788740:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078547.492855:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078547.813170:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078548.517023:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078548.836750:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078549.540807:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078549.860673:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078550.564847:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078550.884732:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078551.589290:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078551.589313:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078551.589319:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078551.589331:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5240 00000100:00080000:0.0:1769078551.589343:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078551.589351:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078551.589397:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007924001408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078551.589431:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1769078551.589431:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078551.589465:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078551.589468:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078551.589473:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5240 00000100:00080000:0.0:1769078551.589478:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078551.589483:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078551.589489:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924001408:192.168.204.143@tcp:250:kworker.0 00000008:00100000:0.0:1769078551.589495:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078551.589511:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007924001536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078551.589523:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078551.589526:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078551.589528:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078551.589532:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5240 00000100:00080000:0.0:1769078551.589549:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078551.589553:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000800:00000100:1.0:1769078551.589564:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000008:00100000:0.0:1769078551.589580:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078551.589622:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007924001664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078551.589622:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078551/real 1769078551] req@ffff969e06a62d80 x1855007924001408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078551.589651:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078551.589655:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078551.589657:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078551.589663:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5240 00000100:00100000:1.0:1769078551.589663:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007924001408/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078551.589670:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078551.589677:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078551.589683:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078551.589683:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078551.589693:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:0.0:1769078551.589698:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007924001792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078551.589701:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924001408:192.168.204.143@tcp:250:kworker.0 00000100:00080000:0.0:1769078551.589716:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078551.589719:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078551.589720:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078551.589725:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5240 00000100:00080000:0.0:1769078551.589728:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078551.589732:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:1.0:1769078551.589737:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001536:192.168.204.143@tcp:38:kworker.0 00000008:00100000:0.0:1769078551.589739:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078551.589752:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007924001920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078551.589822:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078551.589832:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078551/real 1769078551] req@ffff969e06a60700 x1855007924001536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078551.589840:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007924001536/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078551.589922:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078551.589925:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078551.589927:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001536:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078551.589932:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001664:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078551.589947:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078551.589954:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078551/real 1769078551] req@ffff969e06a61500 x1855007924001664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078551.589961:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007924001664/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078551.589967:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078551.589969:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078551.589970:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001664:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078551.589973:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001792:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078551.590252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078551.590261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078551/real 1769078551] req@ffff969e06a60000 x1855007924001792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078551.590268:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007924001792/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078551.590274:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078551.590276:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078551.590278:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001792:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078551.590289:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001920:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078551.590300:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078551.590306:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078551/real 1769078551] req@ffff969e06a63800 x1855007924001920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078551.590312:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007924001920/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078551.590318:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078551.590319:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078551.590321:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924001920:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078551.908741:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078552.612825:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078552.932742:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078553.638520:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078553.956724:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078554.660784:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078554.980709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078555.684753:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078556.004743:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078556.708714:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078556.708726:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078556.708731:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078556.708748:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5245 00000100:00080000:3.0:1769078556.708761:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1769078556.708775:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078556.708790:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:3.0:1769078556.708836:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc380 x1855007924002048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078556.708861:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078556.708864:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078556.708866:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078556.708871:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5245 00000100:00080000:3.0:1769078556.708875:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078556.708880:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078556.708890:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078556.708899:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924002048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:3.0:1769078556.708906:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fd500 x1855007924002176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078556.708918:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078556.708921:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078556.708923:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078556.708927:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5245 00000100:00080000:3.0:1769078556.708930:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078556.708933:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078556.708938:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078556.708948:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fc000 x1855007924002304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078556.708958:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078556.708961:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1769078556.708969:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1769078556.708970:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:3.0:1769078556.708978:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5245 00000100:00080000:3.0:1769078556.708984:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078556.708988:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:3.0:1769078556.708994:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:3.0:1769078556.709005:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7fdc00 x1855007924002432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078556.709012:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078556/real 1769078556] req@ffff969e0a7fc380 x1855007924002048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078556.709018:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078556.709025:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1769078556.709027:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078556.709031:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5245 00000100:00100000:1.0:1769078556.709034:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc380 x1855007924002048/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078556.709036:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078556.709044:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00080000:1.0:1769078556.709048:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1769078556.709050:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00080000:1.0:1769078556.709060:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:3.0:1769078556.709065:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a7ff800 x1855007924002560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078556.709069:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924002048:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078556.709103:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002176:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078556.709139:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078556.709152:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078556/real 1769078556] req@ffff969e0a7fd500 x1855007924002176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078556.709165:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fd500 x1855007924002176/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078556.709173:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078556.709176:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078556.709180:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002176:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078556.709197:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002304:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078556.709213:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078556.709220:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078556/real 1769078556] req@ffff969e0a7fc000 x1855007924002304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078556.709228:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fc000 x1855007924002304/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078556.709233:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078556.709235:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078556.709237:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002304:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078556.709241:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002432:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078556.709252:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078556.709261:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078556/real 1769078556] req@ffff969e0a7fdc00 x1855007924002432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078556.709272:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7fdc00 x1855007924002432/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078556.709281:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078556.709284:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078556.709287:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002432:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078556.709293:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002560:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078556.709311:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078556.709321:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078556/real 1769078556] req@ffff969e0a7ff800 x1855007924002560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078556.709341:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a7ff800 x1855007924002560/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078556.709350:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078556.709380:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078556.709389:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a7ff800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002560:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078557.028666:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078557.732746:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078558.052782:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078558.756865:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078559.076746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078559.781535:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078560.100715:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078560.804824:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078561.124709:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078561.828834:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078561.828846:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078561.828869:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078561.828878:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5251 00000100:00080000:0.0:1769078561.828889:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078561.828895:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00010000:00010000:2.0:1769078561.828921:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078561.828927:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60380 x1855007924002688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078561.828970:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078561.828974:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078561.828976:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078561.828981:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5251 00000100:00080000:0.0:1769078561.828985:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078561.828989:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078561.828999:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078561.829012:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60a80 x1855007924002816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078561.829019:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078561.829021:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078561.829023:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078561.829026:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5251 00000100:00080000:0.0:1769078561.829030:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078561.829034:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078561.829038:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078561.829049:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a63800 x1855007924002944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078561.829067:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078561.829069:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078561.829070:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078561.829074:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5251 00000100:00080000:0.0:1769078561.829077:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078561.829079:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078561.829083:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:1.0:1769078561.829091:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924002688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:0.0:1769078561.829094:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60000 x1855007924003072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078561.829111:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078561.829113:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078561.829114:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078561.829117:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5251 00000100:00080000:0.0:1769078561.829119:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078561.829122:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078561.829124:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078561.829132:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61500 x1855007924003200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078561.829223:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078561.829255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078561.829333:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078561.829357:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078562.148745:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078562.853073:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078563.172696:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078563.876736:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078564.196701:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078564.900774:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078564.900935:0:2405:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.143:988 00000400:00000100:2.0:1769078564.900986:0:2405:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.143@tcp at host 192.168.204.143 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078564.901008:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078564.901053:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078564.901103:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769078564.901112:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078561/real 1769078564] req@ffff969e06a60380 x1855007924002688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1769078564.901113:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000800:00000100:2.0:1769078564.901126:0:2405:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.143@tcp 00000100:00100000:1.0:1769078564.901155:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60380 x1855007924002688/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078564.901167:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078564.901173:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078564.901192:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924002688:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078564.901204:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078561/real 1769078564] req@ffff969e06a60a80 x1855007924002816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078564.901232:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60a80 x1855007924002816/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078564.901241:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078564.901245:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078564.901248:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002816:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078564.901255:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078561/real 1769078564] req@ffff969e06a63800 x1855007924002944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078564.901267:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a63800 x1855007924002944/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078564.901275:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078564.901279:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078564.901282:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924002944:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078564.901311:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078561/real 1769078564] req@ffff969e06a60000 x1855007924003072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078564.901340:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60000 x1855007924003072/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078564.901348:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078564.901352:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078564.901355:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003072:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078564.901362:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078561/real 1769078564] req@ffff969e06a61500 x1855007924003200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078564.901375:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61500 x1855007924003200/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078564.901383:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078564.901387:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078564.901390:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003200:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078565.220746:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078565.941496:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078566.244817:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078566.951572:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078566.951722:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078566.951729:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078566.951733:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078566.951744:0:33176:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5256 00000100:00080000:0.0:1769078566.951771:0:33176:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078566.951778:0:33176:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:0.0:1769078566.951815:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a60700 x1855007924003328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078566.951843:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078566.951847:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078566.951849:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078566.951853:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5256 00000100:00080000:0.0:1769078566.951856:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078566.951859:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078566.951870:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078566.951881:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62d80 x1855007924003456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078566.951890:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078566.951893:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078566.951894:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078566.951898:0:33176:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5256 00000100:00080000:0.0:1769078566.951901:0:33176:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078566.951904:0:33176:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078566.951919:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078566.951929:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a62680 x1855007924003584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078566.951937:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078566.951954:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1769078566.951956:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078566.951960:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5256 00000100:00080000:0.0:1769078566.951964:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078566.951967:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078566.951972:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078566.951981:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61880 x1855007924003712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078566.951990:0:33176:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078566.951993:0:33176:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1769078566.951995:0:33176:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078566.951999:0:33176:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5256 00000100:00080000:0.0:1769078566.952001:0:33176:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078566.952004:0:33176:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:0.0:1769078566.952008:0:33176:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:0.0:1769078566.952016:0:33176:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e06a61180 x1855007924003840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078566.952255:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924003328:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078566.952326:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078566.952356:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078566/real 1769078566] req@ffff969e06a60700 x1855007924003328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078566.952373:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a60700 x1855007924003328/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078566.952384:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078566.952389:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078566.952393:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924003328:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078566.952401:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003456:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078566.952418:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078566.952428:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078566/real 1769078566] req@ffff969e06a62d80 x1855007924003456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078566.952439:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62d80 x1855007924003456/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078566.952447:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078566.952450:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078566.952453:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003456:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078566.952458:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003584:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078566.952473:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078566.952480:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078566/real 1769078566] req@ffff969e06a62680 x1855007924003584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078566.952491:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a62680 x1855007924003584/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078566.952498:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078566.952502:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078566.952505:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003584:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078566.952511:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003712:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078566.952524:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078566.952532:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078566/real 1769078566] req@ffff969e06a61880 x1855007924003712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078566.952543:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61880 x1855007924003712/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078566.952551:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078566.952553:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078566.952562:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003712:192.168.204.143@tcp:8:kworker.0 00000100:00100000:1.0:1769078566.952567:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003840:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078566.952582:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078566.952610:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078566/real 1769078566] req@ffff969e06a61180 x1855007924003840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078566.952623:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e06a61180 x1855007924003840/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078566.952631:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078566.952634:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078566.952638:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e06a61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924003840:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078567.268754:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078567.973214:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078568.292896:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078568.997294:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078569.316728:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078570.020927:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078570.341329:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078571.045636:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078571.366234:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078572.069085:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078572.069774:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) 84979874-d769-417d-a476-16367f366e1a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078572.069783:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078572.069787:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078572.069798:0:24457:0:(import.c:509:import_select_connection()) MGC192.168.204.143@tcp: connect to NID 192.168.204.143@tcp last attempt 5261 00000100:00080000:2.0:1769078572.069811:0:24457:0:(import.c:588:import_select_connection()) MGC192.168.204.143@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078572.069818:0:24457:0:(import.c:623:import_select_connection()) MGC192.168.204.143@tcp: import ffff969e03ab6000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000100:00100000:2.0:1769078572.070207:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007924003968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078572.070373:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924003968:192.168.204.143@tcp:250:kworker.0 00000800:00000100:1.0:1769078572.070447:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078572.070483:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078572.070487:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078572.070489:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078572.070494:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5261 00000100:00080000:2.0:1769078572.070499:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078572.070504:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff969e12057000: import ffff969e14520000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078572.070513:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078572.070529:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007924004096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078572.070818:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078572/real 1769078572] req@ffff969e0a46d880 x1855007924003968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078572.070837:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007924003968/t0(0) o250->MGC192.168.204.143@tcp@192.168.204.143@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078572.070860:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e03ab6000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078572.070864:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078572.070869:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84979874-d769-417d-a476-16367f366e1a:2416:1855007924003968:192.168.204.143@tcp:250:kworker.0 00000100:00100000:1.0:1769078572.071180:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924004096:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078572.071210:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078572.071237:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078572/real 1769078572] req@ffff969e0a46f480 x1855007924004096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078572.071251:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007924004096/t0(0) o38->lustre-MDT0000-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078572.071260:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14520000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078572.071264:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078572.071268:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924004096:192.168.204.143@tcp:38:kworker.0 00000100:00080000:2.0:1769078572.071771:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078572.071776:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078572.071778:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078572.071783:0:24457:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5261 00000100:00080000:2.0:1769078572.071790:0:24457:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078572.071796:0:24457:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff969e12057000: import ffff969e14522000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078572.071803:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078572.071822:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f480 x1855007924004224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078572.072751:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924004224:192.168.204.143@tcp:38:kworker.0 00000800:00000100:1.0:1769078572.072809:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00080000:2.0:1769078572.074568:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078572.074577:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1769078572.074581:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078572.074603:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5261 00000100:00080000:2.0:1769078572.074614:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078572.074622:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff969e12057000: import ffff969e0a427000 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078572.074633:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078572.074665:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46d880 x1855007924004352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078572.074782:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078572/real 1769078572] req@ffff969e0a46f480 x1855007924004224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078572.074805:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f480 x1855007924004224/t0(0) o38->lustre-MDT0001-mdc-ffff969e12057000@192.168.204.143@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078572.074815:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e14522000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078572.074821:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078572.074826:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924004224:192.168.204.143@tcp:38:kworker.0 00000100:00100000:1.0:1769078572.074865:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924004352:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078572.077118:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078572.077156:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078572/real 1769078572] req@ffff969e0a46d880 x1855007924004352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078572.077176:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46d880 x1855007924004352/t0(0) o8->lustre-OST0000-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078572.077188:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a427000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078572.077192:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078572.077197:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924004352:192.168.204.143@tcp:8:kworker.0 00000100:00080000:2.0:1769078572.079628:0:24457:0:(pinger.c:229:ptlrpc_pinger_process_import()) ec62528a-c049-4a35-bee8-59b63d89183b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078572.079639:0:24457:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1769078572.079643:0:24457:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078572.079653:0:24457:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: connect to NID 192.168.204.143@tcp last attempt 5261 00000100:00080000:2.0:1769078572.079681:0:24457:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078572.079689:0:24457:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff969e12057000: import ffff969e0a422800 using connection 192.168.204.143@tcp/192.168.204.143@tcp 00000008:00100000:2.0:1769078572.079702:0:24457:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681347 ocd_grant: 3153920, lost: 0. 00000100:00100000:2.0:1769078572.079733:0:24457:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff969e0a46f800 x1855007924004480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078572.079866:0:2416:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924004480:192.168.204.143@tcp:8:kworker.0 00000800:00000100:1.0:1769078572.080557:0:2416:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.143@tcp 00000100:00100000:1.0:1769078572.080634:0:2416:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078572/real 1769078572] req@ffff969e0a46f800 x1855007924004480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078572.080653:0:2416:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff969e0a46f800 x1855007924004480/t0(0) o8->lustre-OST0001-osc-ffff969e12057000@192.168.204.143@tcp:28/4 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078572.080664:0:2416:0:(import.c:65:import_set_state_nolock()) ffff969e0a422800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078572.080669:0:2416:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.143@tcp failed (-110) 00000100:00100000:1.0:1769078572.080673:0:2416:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff969e0a46f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ec62528a-c049-4a35-bee8-59b63d89183b:2416:1855007924004480:192.168.204.143@tcp:8:kworker.0 00000100:00100000:3.0:1769078572.388793:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078573.094300:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078573.412758:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078574.117091:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078574.436711:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078575.140924:0:32798:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078575.460786:0:32693:0:(client.c:2690:ptlrpc_set_wait()) set ffff969e130996c0 going to sleep for 0 seconds Debug log: 56739 lines, 56739 kept, 0 dropped, 0 bad.